Trusting Remote Users… Can They Identify Problems Without Involving Usability Experts?

Based on our belief that critical incident data, observed during usage and associated closely with specific task performance are the most useful kind of formative evaluation data for finding and fixing usability problems, we developed a Remote Usability Evaluation Method (RUEM) that involves real users self-reporting critical incidents encountered in real tasks performed in their normal working environments without the intervention of evaluators. In our exploratory study we observed that users were able to identify, report, and rate the severity level of their own critical incidents with only brief training.