|
In today’s
environment, there is no guarantee of writing ability based on a level
of education contained in a resume. Asking for previous writing samples
has been shown to be unreliable and very limited in validity. The
writing skills exercise method contained in the tool kit uses
information specifically related to the position. The exercise is
objectively scored without the scorer knowing the identity of the
candidate. The exercise is designed to be used in conjunction with a
structured behavior interview.
Determining the level
of Writing Ability for the Target Job
Your job analysis has
determined that the position requires the ability to write. The question
then becomes “What level of writing ability is necessary upon entry?”
Listed below in ascending level of writing abilities are some examples:
-
Simple Sentence
-
Complex Sentence
-
Paragraph
-
Form Letter
-
Request Letter
response (cut and paste or original)
-
Complaint Letter
response
-
General Information
Correspondence (letter or memo)
-
News Letter Article
-
Press Release
-
Reports
-
Grant Proposals
Developing the exercise
The writing ability
exercise is one type of general exercise known as Work Sample Exercises.
Work Sample Exercises are based on the premise that the best predictor
of future behavior is observed behavior under similar situations. These
exercises require the candidate to perform tasks that are similar to
those that are performed on the job. Thus, the exercise must measure
writing abilities necessary upon entry to the job. The hiring authority
(SME) is responsible for preparing the writing assignment and
instructions for completing the assignment to the candidate. The SME
then completes the writing assignment, which is used to develop the
scoring key.
Developing the scoring
key
Based on the SME’s
completed exercise the scoring key is developed primarily by using the
SME’s judgment and the job analysis. The exercise is usually scored
along two general criteria: 1) format and general use of English and
grammar; and 2) exercise content. Using a Request Letter response (low
to medium ability) as an example the following items could be scored
according to criterion 1: salutation, acknowledgment of letter, letter
closing, signature, grammar, sentence structure, spelling, and word
usage. Using criterion 2: providing the requested materials and
providing additional information could be scored. Rating scales are
used to rate each of the sub-criteria to obtain a numeric score. The
rating scales should be defined by benchmarks for each sub-criterion.
Benchmarks are suggested answers to the questions that are linked to the
rating scale. Benchmarks provide a frame of reference for assessing the
candidate’s responses objectively and consistently. There should usually
be at least three suggested answers for each sub-criterion: a superior,
a satisfactory, and an unsatisfactory response to develop benchmarks,
using the guide below, ask SMEs to create answers that fit the different
levels of the rating scale.
5— What would one
expect or want an outstanding candidate to give as the best possible
answer?
4—
3— What is an
acceptable answer that one would expect a qualified candidate to give?
2—
1— What would one
expect as a poor answer from a candidate who has little or no knowledge
or skill on this job requirement?
Written Exercise
Administration (This is the most critical part
of the exercise)
Depending on the length of the exercise, candidates will
be scheduled before the interview. The written exercise administration
will take place in a room or cube containing a computer with a word
processing program (MS Word). Ideally the written exercise should be
administered in a private and quiet room. Although the actual job may be
performed in a noisy office, every effort should be made to provide the
candidate with a location that is most conducive to completing the
exercise.
An exercise administrator who is not part of the
interview process is necessary to administer the exercise. The exercise
administrator will assign a random ID number to the candidate when they
arrive for the exercise. The candidate will be instructed to use the ID
number at all times during the exercise. The exercise administrator will
then read the specific instructions for the exercise to candidate. When
the candidate has five minutes remaining to complete the exercise, the
exercise administrator will give the candidate a five-minute warning.
When the time limit has expired or the candidate finishes the exercise,
the exercise administrator will collect the candidate’s exercise and
instruction sheet. The exercise administrator will check the candidate’s
exercise to assure only the exercise ID is used on the exercise. The
exercise administrator will then make a photocopy of the original. After
all the candidates have participated in the exercise, the administrator
will retain the originals and provide the copies to the exercise
scorer(s).
Exercise Scoring
It is recommended that the Hiring Authority score all of
the exercises using a score sheet based on the sub-criteria being
scored. If a panel scores the exercise they should be trained. Each
Panel member shall receive a copy of each candidate’s written exercise
and score them independently. After scoring all candidates, the panel
should review their individual ratings as a group and make sure that
there is a general consensus on each sub-criterion. For example, on a
scale of 1 to 5, if one rater gave a 4, one a 3, and the other a 1 on
the same sub-criteria, this might indicate some rater bias or
misunderstanding of the candidate’s comments. All raters should be no
more than one point away from all other raters’ scores. For example, it
would be acceptable to have two 3s and a 2 given on the same question.
If raters find that there is not a consensus on a sub-criterion, they
should discuss the reasons for their ratings and attempt to reach a
consensus. Once the all written exercise scores are final, the exercise
administrator will inform the rater(s) of the names of the candidates.
The results of the interview and exercise may be combined at this point.
|
