The effects of skipping more difficult items on time-limited tests: a quasi-experimental design

Abstract

Standardized aptitude and achievement tests are often accompanied by directions which recommend skipping items perceived as difficult and returning to answer them if time permits. Prior to the present study, the only empirical research in this area was a single study concerning the effects of examinee decisions to follow or disregard these instructions. It was the conclusion of this research that it was to the advantage of high ability examinees to follow the instructions to skip but that conformity to the instructions was disadvantageous to middle and low ability examinees. The present study, in contrast, was concerned with the effects of differing instructions concerning skipping and was quasi-experimental in design. The sample consisted of 423 eighth through twelfth grade students of algebra and biology. Intact classes representing three ability levels were assigned at random to treatment conditions consisting of instructions to skip, not to skip, and with no advice concerning skipping (only “skip” and “do not skip” instructions for the algebra classes.) Schoolwide tests were administered using special answer sheets designed to identify skipped items even when examinees later returned and answered some or all previously skipped. The scores from these tests were used in determination of end-of-course grades. The primary dependent variables were number-right test scores and the number of items skipped. For each subject area, two, two-way factorial analyses of variance evaluated the effect of the differing instructions across ability levels, one for the number-right scores and one for the number of items skipped. For the algebra test, there was no significant treatment effect for number-right scores. However, there was a significant interaction (p≤.01) between ability level and treatment, with high ability examinees receiving “do not skip” instructions performing significantly worse than their counterparts who received “skip” instructions. For the biology test scores, the main effect for treatment and its interaction with ability were not significant (p>.05). For the number of skips in both algebra and biology, the main effect for treatment was significant (p<.01), with the fewest skips in the “do not skip” groups. In addition, for the number of skips in algebra, the main effect for ability was significant (p<.05) with the fewest skips in the middle ability group. While the instructions apparently had their intended effect, and while the treatment-ability interaction in algebra was consistent with the findings of prior research, the number of items skipped by groups instructed to do so were not large, and groups instructed not to skip had meaningful number of skips nevertheless. Thus it was not surprising that neither the biology scores nor the algebra scores differed significantly according to treatment. Extensive secondary analyses of item and other statistics provided no basis for explaining the interaction between ability and treatment for the algebra scores. It was concluded that instructions to skip items had relatively little effect for tests of the type employed in this study.