• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Finally, you can manage your Google Docs, uploads, and email attachments (plus Dropbox and Slack files) in one convenient place. Claim a free account, and in less than 2 minutes, Dokkio (from the makers of PBworks) can automatically organize your content for you.



This version was saved 4 years ago View current version     Page history
Saved by Quan Nha HONG
on April 13, 2017 at 3:45:02 pm

Welcome to the public wiki 'Mixed Methods Appraisal Tool'


Please invite others to use this workspace. Comments and suggestions can be added at the bottom of each page (free comment box). 


Aim of this WIKI: To enable collaborative work for developing a Mixed Methods Appraisal Tool (MMAT).


The MMAT is intended to be used as a checklist for concomitantly appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies).


New complementary information for designing, conducting, and reporting systematic mixed studies reviews: 

Wikitool: http://toolkit4mixedstudiesreviews.pbworks.com

Publication: Pluye, P. & Hong, Q.N. (2014). Combining the power of stories and the power of numbers: Mixed Methods Research and Mixed Studies Reviews. Annual Review of Public Health, 35, 29-45. Complimentary online access:  http://arjournals.annualreviews.org/eprint/qFxpDWrNzjzwjfkgtd4V/full/10.1146/annurev-publhealth-032013-182440 (Note from the journal: ‘Multiple distribution, publication or commercial usage of this copyrighted material requires submission of a permission request addressed to the Copyright Clearance Center (http://www.copyright.com/)').


The MMAT is a tool in development, and must be used with caution. The development of the MMAT is supported by a project called ‘Content Validity, Usability and Reliability of a Mixed Methods Appraisal Tool (MMAT)’ (including workshops, presentations and fellowship/prize/grant from the Canadian Institutes of Health Research - CIHR).


For instance, you may state that the Mixed Methods Appraisal Tool is:

  • Designed for systematic reviews that include qualitative, quantitative and mixed methods studies;
  • Efficient as it allows to use one tool for concomitantly appraising the most common types of empirical studies;
  • Addressing the quality of mixed methods studies (appraisal of qualitative, quantitative and mixed methods components);
  • Based on a constructionist theory and a literature review;
  • Content validated using feedback from experts and workshops;
  • Pilot tested for reliability;
  • Already used worldwide for at least 50 systematic mixed studies reviews (to our knowledge). 


Current version: The 2011 version of the MMAT is available here (criteria and tutorial)

Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., Boardman, F., Gagnon, M.P., & Rousseau, M.C. (2011). Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Department of Family Medicine, McGill University, Montreal, Canada. Retrieved on [date] from  http://mixedmethodsappraisaltoolpublic.pbworks.com. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ


Your feedback is important for us. If you are using the MMAT for a systematic mixed studies review, please let us know : pierre.pluye@mcgill.ca or quan.nha.hong@mail.mcgill.ca


French version

The 2011 version of the MMAT was translated into French by Dr Emilie Robert: click here to download the French version.  Please note that this version has not been validated yet. 


FAQ (last update: 2017-04-13) We added a page FAQ based on questions received from MMAT users. 


2011 version (efficiency and reliability testing)


This NEW presentation suggests the MMAT-v2011 is an efficient tool, while its reliability needs further improvement, particularly for two items of the qualitative research domain, which includes the sentence ‘appropriate consideration’ (items 1.3 'Is appropriate consideration given to how findings relate to the context, e.g., the setting, in which the data were collected?' and 1.4 'Is appropriate consideration given to how findings relate to researchers’ influence, e.g., through their interactions with participants?'). We noted that the independent reviewers understood this sentence in a different manner. A reviewer considered that ‘appropriate consideration’ was given when there were at least few details, whereas the other reviewer looked for a detailed description of specific strategies. In some articles reporting qualitative research, neither the aspects corresponding to these items are mentioned; other articles mention them, but with few details (e.g., a simple description of the investigators’ experience), while articles with detailed descriptions of these aspects are rare (e.g., a strategy used to document the influence of the researchers such as a reflexive diary). This leads us to suggest discrepancy in reviewers’ interpretation of these items can be resolved by reviewers establishing a common understanding of these two items prior to beginning  the critical appraisal. Alternatively, authors can be contacted when these criteria are not met, as suggested in the MMAT tutorial.


Pilot version (efficiency and reliability testing)


Initial version (content validation)

  •  Pluye P, Gagnon MP, Griffiths F & Johnson-Lafleur J (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4):529-546. 


Limitations & strenghts

  • The MMAT is not a guidance for writing and reporting mixed methods studies such as GRAMMS 'Good Reporting of a Mixed Methods Study' (O'Cathain et al., 2008), and does not permit a comprehensive evaluation of mixed methods studies such as the conceptual framework proposed by O'Cathain (2010).
  • Crowe and Sheppard (2011) reviewed critical appraisal tools, and found only one tool that addresses the quality of mixed methods studies: the MMAT. They suggest the MMAT is in the top-five tools with respect to the explanation of the development of the items, and the presence of a tutorial. They mentioned the content validation of the MMAT, but were not aware (at the time of the publication) that the pilot version of the MMAT was being tested for reliability.
  • The MMAT has been reviewed by the National Collaborating Centre for Methods and Tools (NCCMT): See http://www.nccmt.ca/resources/search/232. E.g., this evaluation states that the MMAT 'is well suited to a public health context, particularly for questions related to complex interventions that are context-dependant and process-oriented.' 


Suggestion for using the MMAT

Based on discussions with MMAT users, we suggest the following idea.

Use MMAT results to create 2 groups of similar sizes (similar number of studies in each group) that you can compare (similar to a 'sensitivity analysis'): lower vs. higher MMAT studies.

State that you included all relevant studies, some with a lower MMAT result (for different reasons) and others with a higher MMAT result, which allows a comparison between lower vs. higher MMAT studies.

You can repeat such comparison with other study characteristics such as country, year, etc.

This offers a rationale to compare two subsets of studies.

If results are similar in the 2 sets, your  synthesis results will be more valid (if QUAN synthesis) or trustworthy (if QUAL synthesis).

Finally this avoid blaming low-MMAT studies as the MMAT assessment is not an objective measure.



  • Crowe, M., & Sheppard, L. (2011). A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed. Journal of Clinical Epidemiology, 64(1), 79-89.

  • O'Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy, 13(2), 92-98.

  • O'Cathain, A. (2010). Assessing the quality of mixed methods research: towards a comprehensive framework. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 531-555). Thousand Oaks: Sage.

  • Pluye, P., Bush, P., Macaulay, A., Khanassov, V., Queiroga, R., Loignon, C., et al. (September 2013). The Mixed Methods Appraisal Tool for assessing studies with diverse designs: Example from a systematic mixed studies review on the key processes and outcomes of Participatory Research with Health Organizations. Annual International Cochrane Colloquium, Quebec City. 

  • Pluye, P. & Hong, Q.N. (2014). Combining the power of stories and the power of numbers: Mixed Methods Research and Mixed Studies Reviews. Annual Review of Public Health, 35, 29-45


Visitors (since March 12, 2013):


Comments (0)

You don't have permission to comment on this page.