An investigation of multi-attribute utility technology (MAUT) as an evaluation method in an organizational training environment

Abstract

Multi-Attribute Utility Technology (MAUT) was investigated as an alternative organizational training evaluation method. Research questions focused on what problems emerge in involving stakeholders in training evaluation, what technical problems emerge in the application of the MAUT steps, and what can be learned about the credibility of MAUT results and their impact on decision making.

Five employee training courses offered by a city government training unit were evaluated. Two groups of stakeholders developed desired course attributes and indicators using MAUT. Survey data collected from students and their immediate supervisors provided indicator measurements. Location measures and utilities were provided to stakeholders and students. Stakeholders then evaluated MAUT from the participant perspective.

Positive and negative findings had implications for the effective use of MAUT. Positive findings were that stakeholders valued the group process and opportunity to have input to the evaluation; surveys were useful for measuring attribute indicators; the development of attributes, indicators, weights and location measures presented no technical problem; and, that MAUT results were credible to students and Stakeholders. Negative findings were that stakeholder availability was very difficult to obtain; not all stakeholders could deal with the conceptual tasks of MAUT; surveys were not feasible for attribute development; calculation and interpretation of MAUT utilities were seen as too complex by stakeholders; and, the use of MAUT results to influence decisions was hindered by the perceived complexity and the absence in the study of the primary decision maker for training.

It was concluded that a major strength of MAUT is that it affords the investigator the opportunity to easily tap into the organizational culture and political setting of a program. In addition, the strengths of MAUT as a training evaluation method are highly dependent on inclusion of all relevant stakeholders, a strong commitment of stakeholder time, stakeholder ability to communicate values and priorities, and, the clear and simple reporting of MAUT results. Detailed recommendations for the effective use of MAUT are provided.