A Preliminary Rubric Design to Evaluate Mixed Methods Research

Abstract

With the increase in frequency of the use of mixed methods, both in research publications and in externally funded grants there are increasing calls for a set of standards to assess the quality of mixed methods research. The purpose of this mixed methods study was to conduct a multi-phase analysis to create a preliminary rubric to evaluate mixed methods research articles. This study included four research questions:

1. What are the common evaluation criteria found in the contemporary methodological literature pertaining to the design of mixed methods research?

2. What evaluation criteria do experts in the field of mixed methods research perceive as the most important when distinguishing top-quality research in mixed methods?

3. What differences are there in the outcome of the rubric for evaluating mixed methods research identified from the literature compared to those advocated most uniformly by a panel of mixed methods research experts?

4. What are disciplinary differences between the use of mixed methods and views about evaluating it, including the role of paradigms in mixed methods research?

In the first phase of this multi-phase mixed methods study I used an inductive qualitative process to identify the quality criteria endorsed by 12 methodologists with a long-term involvement in mixed methods research. In the second phase of this study I conducted a quantitative analysis to pilot test a set of criteria identified in the qualitative phases. The sample for both phases of this study was comprised of the same eight males and four females from multiple nationalities. Respondents to the on-line survey rated all 14 items as being important, with 11 of the 14 items being rated as very important or higher.

When considered together, findings from the two phases of this study provide a interesting view of attitudes about the use and application of quality standards to the mixed methods literature. While there was agreement about what elements were important to evaluate, there was not an agreement about the idea that one set of standards could be applied to all mixed methods studies.