Computer-assisted item and test pre-analysis: a new direction in qualitative methods

Abstract

To date, the major emphasis in test and item evaluation has been directed toward statistical post-measures which rely heavily on data gathered from the administration of the instrument. These primarily summative techniques are limited, however, in that they are incapable of providing information about and item/test before it has been sent for field trials. This research presents a new direction in test and item analysis which, using test/item writing heuristics, provides a previously unavailable technology for instrument pre-analysis. The new field of "qualitative item and test pre-analysis" is proposed and described. The implications to the field are discussed in addition to specific suggestions for the use of this new technology.

The design and creation of a base-case item and test pre-analysis expert system (ITAX) is also detailed, including the heuristics incorporated, implementation methodologies and limitations. The heuristics incorporated into the system include the detection of: two varieties of grammatical cues, negation/multiple negation, repetition of phrases within an options list, presence of too few options, inconsistent length of distractors, use of all- and none-of-the-above, repetition of significant words from the stem to the options, randomness of multiple choice answer placement, balance of true/false items and length of true/false items. A comprehensive reference to the system is also provided.