An empirical test of the assumptions of processing invariance in laboratory studies of performance appraisal

Abstract

Laboratory studies of the cognitive processes of performance appraisal which employ undergraduates as raters necessarily imply one of two assumptions of processing invariance: the constant category assumption or the constant familiarity assumption. The constant category assumption is implied in studies having undergraduate students rate unfamiliar occupations, and then generalizations are made to supervisors in organizational settings who are much more familiar with the job they are rating. On the other hand, the constant familiarity assumption is implied in studies in which generalizations of performance ratings are made only when raters rate an occupation with which they are familiar (i.e., when students rate teaching). The present study tested these two competing assumptions by varying job familiarity and appraisal purpose in a 2 (rater population) X 2 (target occupation) X 2 (appraisal purpose) X 3 (performance) mixed factorial design. 40 professional carpenters recruited from various contracting firms in the Southwest Virginia, and 40 undergraduate college students from Virginia Polytechnic Institute viewed videotaped performances of three carpentry students performing four different woodworking tasks, and three teaching assistants giving brief lectures. Appraisal purpose was manipulated orthogonally by telling half of the subjects to form a general impression of the ratees’ performances (impression-set) and telling the other half to remember as many of the behaviors and actions of the subjects as possible (memory-set). Job familiarity served as a repeated measure, and was manipulated by crossing rater population (student vs. carpenter) with target occupation (teaching vs. carpentry). It was predicted that subjects familiar with the occupational category they assessed would (a) vary their processing strategies according to appraisal purpose (i.e., recall more judgments under an impression-set and recall more behaviors under a memory-set); (b) better recall the order of ratee performance information; (c) better discriminate between ratee performance levels; and (d) provide more accurate ratings than unfamiliar raters. Analysis of subjects’ free recalls generally failed to support the hypotheses, partly due to the failure of the appraisal purpose manipulation. For outcome measures, results provided partial support for the hypotheses in that job familiarity led to significant differences in performance discrimination and rating accuracy only when subjects rated the carpentry occupation. No differences were seen when subjects rated teachers. While these findings tend to provide greater support for the constant category assumption than for the constant familiarity assumption, some problems with the use and development of the teaching videotapes may have exacerbated these effects. Implications for future performance appraisal research and the application of performance ratings are offered.