An investigation of the relationships of test characteristics and personality variables to partial information and misinformation in multiple-choice test scores

The purpose of this study was to determine the relationships between selected measures of personality and various score components obtained from a multiple-choice test administered under a response mode proposed by Coombs in 1953. Previous studies using response modes and scoring rules other than the conventional number-right procedures have generally revealed increases in test score reliability and little change in criterion-related validity. The rationale offered for most of these alternative procedures is that the direct inclusion of partial information in test scores provides additional information about levels of examinee knowledge and, therefore, ought to enhance test characteristics. Alternatively, however, the observed increases in test reliability may result from the introduction of a reliable but extraneous source of score variance associated with personality factors evoked when examinees are required to express their assurance about each answer. This study attempted to determine whether the reliability increases accompanying the use of the Coombs mode were due to personality contamination of the scores or to other characteristics of the response mode.

The examinees, 278 teacher trainees in a U.S. university, completed several personality measures and also answered an English achievement test with (1) the Coombs mode, (2) the conventional number-right responses, and (3) an open-ended response format. Their responses were grouped, based on three different penalty conditions for wrong responses. Various total test, information, and misinformation scores were calculated from these responses.

Reliability estimates for test scores under the Coombs mode were higher for all three groups than the estimates for number-right scores. Validity coefficients were unchanged or lower. Multiple linear regressions of the personality variables on the total scores indicated that there was no unique involvement of personality variables in the Coombs scores beyond that also present in number-right scores and in empirical choice-weighted scores. Thus, in these groups of examinees, Coombs directions had a pervasive effect on test responses that did not explain the reliability and validity changes that accompanied the Coombs procedure.

Since the added information about levels of examinee knowledge, assumed to be included in the Coombs scores, was in the form of direct credit for partial information and penalty for misinformation, explanation for the reliability and validity changes was sought in these components. Various rescoring procedures were used to isolate the effects of these components on reliability and validity estimates. From these analyses, it was determined that reliability was enhanced by the Coombs mode to the extent that the guessing components in the scores were reduced or eliminated. The reduction of guessing was a function of the opportunity provided by the Coombs mode for examinees to express all bits of partial information. Validity was decreased to the extent that removing guessing reduced the similarity of the test to the validity criteria. Misinformation was found to have little effect on either reliability or validity. These results suggest that, under some circumstances, the Coombs mode may provide increased reliability without the personality contamination that is present in other alternative response modes.