Comparing 12 finite state models of examinee performance on multiple-choice tests

Finite state test theory models the response behavior of an examinee and establishes the relationship between the ability of the examinee and the observed responses on a multiple-choice test. In finite state modeling, various assumptions about item characteristics and the examinees’ response strategies are made to estimate an examinee ability, and willingness to guess.

Twelve sets of plausible assumptions about identifiability of distractors and examinee guessing strategies were adopted and the corresponding finite state models were actualized. Three consequences of the adoption of the 12 sets of assumptions were investigated: 1) the extent to which the resulting ability estimates rank ordered the examinees similarly, 2) variation in the magnitude of ability estimates and the estimated willingness to guess across the 12 models, and 3) the extent to which conclusions about examinees subgroups would differ according to the model employed. Also, conventional number-right scores were compared with the finite state scores with respect to the three outcomes just listed.

All scoring methods rank ordered the examinees essentially the same. The magnitude of the finite state scores varied considerably across models mainly due to differing assumptions about the identifiability of distractors. Differing assumptions about examinee guessing strategy had surprisingly little effect on the magnitude of the ability estimates, though estimates of willingness to guess varied consistently according to the assumed strategies. Conclusions about group differences also varied across the models as a result of differing assumptions about both item characteristics and examinee guessing strategies.