MIXED METHODS APPRAISAL TOOL: STRENGTHENING THE METHODOLOGICAL RIGOR OF MIXED METHODS RESEARCH STUDIES IN NURSING

of Nursing. Greensboro, North Carolina, Estados Unidos. ABSTRACT Objective: to describe the use of the Mixed Methods Appraisal Tool to appraise and to strengthen the methodological rigor of mixed methods studies in nursing. Method: a theoretical essay was used to describe the application of Mixed Methods Appraisal Tool to support the development and assessment of mixed methods research in nursing. Four mixed methods articles in nursing were purposely chosen and evaluated based on the Mixed Methods Appraisal Tool criteria. Results: Mixed Methods Appraisal Tool is a tool for evaluating primary mixed methods studies based on five evaluation criteria: 1) justification for adopting mixed methods methodology; 2) integration between the quantitative and qualitative components; 3) interpretation of integrated findings of quantitative and qualitative data; 4) presentation of divergences between quantitative and qualitative results; and 5) compliance with the methodological rigor of each individual approach in mixed methods research. Conclusion: Mixed Methods Appraisal Tool is an instrumental resource that can be used to appraise mixed methods research and strengthen the methodological rigor in planning and Protocolo de investigación. Investigación en enfermería. Métodos. Estudios de evaluación como asunto. Enfermería.


INTRODUCTION
To strengthen the methodological rigor of scientific research and robust publications, guidelines and protocols for the planning, execution and evaluation of research studies have been increasingly developed and used by various disciplines, including health and nursing. Examples of guidelines include the International Committee of Medical Journal Editors (ICMJE) 1 and Enhancing the Quality and Transparency of Health Research (EQUATOR network) 2 . Due to the insufficient reporting of study findings in the literature, the international EQUATOR network was established 15 years ago with the aim of promoting higher quality and more transparent reporting of different study designs 2 . To date, the network comprises 463 reporting guidelines in the form of checklists, including guidelines for randomized clinical trials 3 , observational studies (cohort, case-control and cross-sectional) 4 , systematic reviews and meta-analyses 5 , qualitative studies 6 and other study types 2,7 .
These guidelines can help increase robust and transparent reporting on the production of scientific knowledge, which is of critical interest to researchers/authors, editors and reviewers of scientific journals. In general, the guidelines require reporting with clarity and accuracy for the title, research problem, state of the science, study objective(s) and design, method and so on 3-7 . In the areas of health and nursing, the use of recommendations for the production of research reports is grounded in the context of evidence-based practice based on scientific and rigorous work that leads to knowledge translation by professionals. Yet, a recent literature review of research studies from seven countries revealed that nurses in the hospital context still have insufficient knowledge to critically appraise scientific evidence and difficulties in implementing evidence-based practice 8 .
In some parts of the world, there is a tendency for nurses to favor studies that are grounded in positivism -a fact that can be explained by the strong influence of the biomedical model in the profession [8][9] . However, it is known that the plurality of scientific research in nursing is real. Therefore, nurse researchers may use diverse theoretical-methodological approaches based on the study objectives to generate new knowledge 10 .
Within the range of possibilities for scientific research in nursing lies the field of mixed methods research, which is made up of both quantitative and qualitative designs 11 . In this sense, mixed methods research has emerged as a new scientific paradigm which helps us to deepen our understanding about complex health issues for the betterment of human beings 10,12 . We recommend that, if reasonable and possible, nurse researchers opt for mixed methods research when examining and intervening in complex social and health problems. Mixed methods research, even though can be laborious and challenging, its findings can promote a richer quality of life and better care for our clients, lead to the increased visibility of the nursing discipline, and advance nursing science overall 10,13 .
Although there has been an increase in the use of mixed methods methodology in nursing, the research process including data integration and findings dissemination are, in general, still embryonic among nurse researchers in many parts of the world 9,14 . Evidence shows that some mixed methods studies in nursing seem to have methodological weaknesses, including a lack of clarity and insufficient information as to the purpose; research design/justification; weights assigned to the quantitative and qualitative arms and data integration strategies 9,14 .
Therefore, it is considered that the correct application of the mixed nursing research methodology is a challenge for researchers in the field. Thus, there is a need to increase means/strategies/resources that enable actions for the rigor and scientific soundness of research with this innovative methodological paradigm. One possibility for this is the adoption of the Mixed Methods Appraisal Tool (MMAT), a tool that outlines the object of this study and that can be useful in strengthening the methodological rigor of mixed methods studies 15 .

4/13
This study aimed to describe the use of the Mixed Methods Appraisal Tool to appraise and to strengthen the methodological rigor of mixed methods studies in nursing.

METHOD
This is a theoretical-reflective essay, which involved problematizing the theme, reading and theoretical abstraction of technical-scientific works on mixed methods of research. It is noteworthy that the team of authors of the study was composed of researchers with experience in the use of mixed methods in Nursing research.
This study involves a description of MMAT in three areas: 1) Mixed Methods Appraisal Tool: 1) origin and characteristics; 2) appraisal criteria; and 3) its application to critically appraise the methodological rigor. For the development of the item 3, four mixed research studies developed in Nursing were selected intentionally to exemplify the use of the tool. For this selection, methodological rigor, originality and topicality were considered.

MIXED METHODS APPRAISAL TOOL: ORIGIN AND CHARACTERISTICS
MMAT is an assessment tool for scientific studies developed in 2006 16 by researchers from Canada (McGill University, Department of Family Medicine) and the United Kingdom, linked to colleges/ departments in the health area, including Nursing. It was reformulated 17 and updated 15 after a literature review and critical analysis carried out by experts, using the Delphi technique 18 . The previous version of MMAT dates back to 2011 and is translated and cross-culturally adapted to the Brazilian reality 19 . However, the most current option of the tool is from 2018 15 and is available in English and French. All versions have unrestricted open access.
Originally, MMAT was developed to assess the quality of primary studies recruited from mixed literature review research studies [15][16][17] , since it includes research evaluation criteria from five methodological categories: randomized clinical trials; non-randomized clinical trials; observational quantitative studies (cross-sectional, case-control and cohort), qualitative studies, and mixed-methods research 15 . It is therefore assumed that MMAT can be a useful tool for investigative planning, in addition to its evident purpose in evaluating primary studies.
Although MMAT was developed with the main purpose of helping to conduct systematic reviews 15,17,19 , the authors encourage the use of additional assessments, such as those already mentioned according to the EQUATOR network 2 . In addition, it is recommended that the evaluation is performed by two researchers/reviewers, independently, and that they have experience in the development of the study designs to be appreciated 15 .
In its structuring, in addition to introductory/conceptual elements, an explanatory diagram for defining the type of study to be evaluated and the technical-scientific basis of the tool, MMAT 15,17 has two parts. The first consists of the evaluation form itself, presented as a checklist. In this part, there are six "blocks", of which the first refers to "filter questions", which are applied to the the five (all) methodological categories of the studies that can be appreciated by the tool. In turn, the other five components are specifically related to study designs to be evaluated, in the following order: qualitative studies, randomized clinical trials, non-randomized clinical trials, descriptive quantitative studies (observational epidemiological -cross-sectional, cohort and case-control), and mixed-methods research 15,17 .

5/13
In the current version, for each methodological category of studies evaluated by MMAT, five check items are applied, which determine compliance with the established evaluation criteria. These items have options for answering yes, no or indeterminate, in addition to a space for comment(s) by the evaluators 15 .
MMAT as a whole has 25 specific evaluation items/criteria (five from each methodological modality), in addition to two previous questions that are common to all types of study, that is, each study is subjected to the evaluation of seven items/criteria 15 . In the practice, this means that each of the five methodological categories that can be evaluated (that is, each study) is subjected to five evaluation criteria specific to this methodological category, in addition to two questions applicable to any of these categories. In addition to that, it is important to emphasize that the authors of MMAT suggest that, in case of a negative or indefinite answer to one or to the first two general questions, it is to be considered that the research is not eligible or susceptible to be appreciated by the tool 15,17 .
The general items contained in MMAT concern the clear existence of definition of the research question(s) and/or objective(s), as well as whether the data presented by the study are sufficient to answer these questions 15,17 . It is interpreted that this may mean that, by not stating a research purpose or by enunciating it vaguely/inconsistently, in addition to not having information applicable to the answers to a certain purpose, the study should not be evaluated, because it does not have enough empirical information to confer it scientific research status.
The specific items/questions to the evaluation of each methodological category in MMAT are naturally linked to the means that boost or ensure the scientific-methodological rigor of each study design 15 . Despite recognizing the importance and the potential contribution of each of the evaluation possibilities by the tool, it is prudent to reassert that the central object of this study is to debate MMAT in the specialized context of mixed-methods research studies, envisioning potentialities in addition to being an evaluative tool, but also a potential contribution to the rigorous planning of mixed studies, especially in the area of Nursing.
After duly evaluated, each study submitted for consideration by MMAT can obtain a more quality complete and accurate assessment of its methodological. This is illustrated in a systematic mixed-methods review that assigns proportions of conformity to the quality to the studies evaluated 20 , according to the 2011 version of MMAT 17 .
It is worth noting that the current version of MMAT 15 incorporated two questions into the specific evaluation block for research using mixed methods compared to the previous version, which has three questions/items for this study design 17 . It is believed that this signal increases in the criteria for defining the methodological quality in this recent scientific paradigm, which is important and necessary, since difficulties and even inconsistencies in its proper conduction are recognized [10][11]14 . Thus, the knowledge of criteria that assist in the methodological rigor of mixed-methods research is of interest for the Nursing science.

MMAT CRITERIA FOR EVALUATING MIXED METHODS RESEARCH STUDIES
For the purpose of the present article, we focus solely on five specific criteria to appraise the methodological rigor of mixed methods studies at MMAT: 1) "Is there an adequate rationale for using a mixed methods design to address the research question?" (justification); 2) "Are the different components of the study effectively integrated to answer the research question?" (integration); 3) "Are the outputs of the integration of qualitative and quantitative components adequately interpreted?" (interpretation); 4) "Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?" (disagreements); and 5) "Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?" (adherence) 15 . The response options of "yes", "no" or "can't tell" are the same as those of the two screening questions 15 . It is worth noting 6/13 that the current version of MMAT 15 has two more questions/criteria than the 2011 version, an effort to increase the methodological rigor for mixed methods research appraisal as a way to recognize and address methodological and reporting inconsistencies and inadequacy [10][11]14 .
MMAT Criterion 1: Justification -"Is there an adequate rationale for using a mixed methods design to address the research question?" 15 When appraising a mixed methods study based on this criterion, the reviewer should look for the rationale for the needs for both quantitative and qualitative arms. The rationale should be clear and justifiable based on the state of the science and gap identifications. In general, there are two main reasons for using mixed methods research: complementarity and sequentiality 11 . Complementarity is used to provide a more complete story of the investigated phenomenon. In other words, quantitative data are complementary to qualitative data, or vice versa; with only one type of data, the phenomenon cannot be fully understood 21 . For sequentially, one research approach comes before the other approach based on the research questions and the needs to fill the gaps for the phenomenon of interest. The qualitative arm can come before or after the quantitative arm, or vice versa, again, based on the state of the science and the gap identification 11 .
MMAT Criterion 2: Integration -"Are the different components of the study effectively integrated to answer the research question?" 15 In mixed methods research, a basic premise is the need for a clear integration between quantitative and qualitative results, as opposed to presenting research results in each arm separately 13 . Data integration, a crucial aspect in mixed methods research, is defined as a clear presentation of the association between the quantitative and qualitative arms 12,15 . The decision on how integration should take place depends on the study objectives, design, data collection and analysis strategies, as well as the researchers' experiences and skills. In sequential studies, for example, the integration may occur during the data analysis in the first phase that guides the data collection process in the second phase 22 . Data can also be merged especially in convergent mixed methods studies 11 . Creswell 23 classified integration types in mixed methods into four categories: data merging, data explanation, data building, and data embedding: y Data merging: qualitative and quantitative data are analyzed separately, then findings from both arms are compared for differences and similarities in a convergent design 24 .
y Data explanation: qualitative findings help explain quantitative findings in an explanatory sequential design 22 .
y Data building: qualitative findings inform future quantitative research questions and design in an exploratory sequential design.
y Data embedding: qualitative findings can be embedded and help to explain quantitative findings in an intervention design 21 .
Strategies used to display data integration may include a joint-display matrix and a visual representation of quantitative and qualitative findings. Such strategies can facilitate clear presentations of findings in both arms and are more visually attractive, displaying how the findings are compared, connected, embedded, and interrelated. In some cases, data conversion is used-transforming qualitative data into quantitative numerical data, or vice versa 11,22 .

7/13
Recently, authors 25 from the United Kingdom proposed a step-by-step model for data integration in mixed methods research, the Pillar Integration Process (PIP). PIP comprises four stages: listing, correspondence, verification, and construction with the end product as a joint display that clearly represents the integration of qualitative and quantitative findings 25 .

MMAT Criterion 3: Interpretation.
"Are the outputs of the integration of qualitative and quantitative components adequately interpreted?" 15 After the integration of qualitative and quantitative results, the findings need to be interpreted based on the similarities and divergences (disagreements, discrepancies or dissonances). A successful integration should result in a view closer to the integral of the phenomenon of interest, rather than the sum of the parts (findings) from each arm in isolation 11,[23][24] .
MMAT Criterion 4. Disagreements -"Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?" 15 For this criterion, when there are no disagreements between quantitative and qualitative results, the reviewer rates "Yes". However, if disagreements arise, to rate this criterion as "Yes", a clear explanation is needed as to how such disagreements are handled and interpreted 15 .

MMAT Criterion 5.
Adherence -Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?" 15 This criterion involves an appraisal of the research method on the quantitative and qualitative arms, separately. To evaluate the quantitative arm, use MMAT Part I: study types 2 (randomized controlled trails), 3 (quantitative non-randomized) or 4 (quantitative descriptive), accordingly and use study type 1 for the qualitative arm 15 . Both arms need to meet completely the criteria of scientific rigor to be considered of good quality for the mixed methods study 15 . When both arms are rated different for the level of quality, the overall quality of the mixed methods study must be assigned to the lower rating 15 . For instance, if the quantitative arm is rated with "high" quality while the qualitative arm is rated as "low" quality, the overall quality of the mixed methods study must be "low".
In sum, MMAT can be used to appraise a mixed methods study using two screening questions and additional questions based on the research questions and the types of the quantitative and qualitative arms. The integration in a mixed methods study must take a holistic approach, occurring at some point during the research process: data collection, data analysis, and when findings emerged 22 . Such an integration should result in meta-inference, generating a "complete" story of the phenomenon of interest [22][23] . Instead of conducting two or more separate studies on one topic, applying mixed methods research to generate meta-inferences can be valuable, critical for nurses whose clients' health and life circumstance are usually complex which also demand complex answers 10,15 .

APPLICATION OF MMAT TO APPRAISE METHODOLOGICAL RIGOR OF MIXED METHOD STUDIES IN NURSING
Across the globe, even though researchers in different disciplines have increasingly acknowledged the benefits of mixed methods research, some nurse researchers still subscribe to either quantitative or qualitative research methodology because their methodology of choice is aligned with their perspective/ vision into the research problem 10 .

8/13
Planning and conducting mixed methods research can be laborious and challenging because both qualitative and quantitative arms must be well planned, executed, and integrated. If one arm's methodological rigor falls off, the other arm and the overall study may suffer. In this section, we purposefully selected four mixed methods studies based on their originality and common topical area on nurses' perceptions in different fields: one from Australia, two from Brazil, and one from Denmark [26][27][28][29] . We demonstrate the use of MMAT 15 in a table format to appraise the selected mixed methods studies. Due to space limitations, we only appraise mixed methods criteria (1-4), omitting criterion 5 that requires an appraisal of the methodological rigor in the qualitative and quantitative arms separately 15 .
Study 1: researchers in Australia used an exploratory sequential mixed methods design to explore and describe nurses' attitudes and behavior towards patients' use of complementary therapies. They collected qualitative data in the first phase from 19 professionals through semi-structured interviews. Based on the themes emerging from this stage, a quantitative research instrument was developed and administered online to 614 nurses across the country. For data integration, data building was used in which the results of the qualitative interview were applied to establish the questions asked in a national survey 26 .
Study 2: researchers in southern Brazil used an explanatory sequential mixed methods design to examine the association between accreditation and nursing team's satisfaction. In the first phase (quantitative correlational cross-sectional), the Brazilian version of the Index of Work Satisfaction was administered to 226 nursing staff at three general hospitals (one private/certified in excellence by the National Accreditation; one private/non-certified; and one public/non-certified). Results showed that nurses from the certified hospital were more satisfied with their work than those from non-certified hospitals. The second phase (qualitative) complemented the quantitative phase through interviews (n=39). Data were analyzed by Collective Subject Discourse (CSD) technique. Through the use of a joint display, the CSD findings were presented in a table format with the quantitative results, which were enriched and supported by the qualitative findings 27 .
Study 3: researchers in Brazil explored healthcare professionals' resilience levels and processes in caring for people with mental disorders in a psychiatric hospital. A convergent mixed methods design was used. In the quantitative cross-sectional arm, the Resilience Scale was administered to 40 health professionals, most of whom were nursing technicians, followed by physicians, and registered nurses. In the qualitative arm, Grounded Theory was applied (n=26) using initial and focused coding. Quantitative data showed that nursing clinicians had the lowest level of resilience compared to other groups. Younger healthcare professionals with fewer years of work experience had higher scores than those who were older with more years of work experience. Qualitative data enriched resilience quantitative findings in this study 28 .
Study 4: researchers in Denmark used a convergent mixed methods design to examine the effect of the "Rehabilitation 24/7" education intervention on nursing staff's ability, opportunity and motivation to work in a rehabilitation setting. Quantitative data were collected pre-and post-education (n=40). Qualitative data (n=10) were analyzed by thematic analysis, guided by an existing model of ability (physical and psychological), opportunity (physical and social) and motivation (automatic and reflective). Data integration was carried out during the discussion process. The authors concluded that based on qualitative findings, the educational intervention strengthened the nursing staff's knowledge and beliefs about patient rehabilitation, in addition to increasing awareness of their own role 29 .

1) Justification
Application of qualitative arm for themes generation and quantitative arm for instrument development to examine the phenomenon of interest.
Gaining a deeper understanding of the phenomenon investigated.
Identifications of convergences and differences between qualitative and quantitative results.
Use of both methods combined to gain a better understanding about the education program.

2) Integration
A quantitative questionnaire was developed based on the qualitative results in phase 1.
Results from quantitative arm were compared and contrasted to qualitative results. A joint display was used to present the integration of the final results.
A diagram presented to compare between quantitative and qualitative results.
Data were integrated in the discussion section as a diagram, comprising the methodological path and final findings.

3) Interpretation
To justify the evidence using triangulated qualitative and quantitative data to improve the findings.
Qualitative findings supported quantitative findings about staff's satisfaction working for an accredited hospital.
Younger healthcare professionals with fewer years of work experience had higher scores than those who were older with more years of work experience. Qualitative data enriched resilience quantitative findings.
Quantitative results showed no major improvements after the education program. Qualitative results helped enrich the understanding of the changes in the nursing team regarding the members' behaviors after the intervention.

4) Disagreements
Results of the quantitative survey on the "Integrating complementary therapy" item were somewhat consistent with results in the qualitative arm.
Results in both quantitative and qualitative arms showed similarities in that nurses working for an accredited hospital were more satisfied with their work than those working for non-accredited hospitals.
Factor III in the resilience scale had the lowest score when compared to other factors. Qualitative results showed that healthcare professionals constantly searched for positive adaptations to the work situations.
The quantitative results did not show any significant change after the intervention. However, qualitative results revealed immediate unit structural changes after the educational program.
In sum, the appraisals of the selected articles presented in Chart 1 show that mixed methods research can enrich the understanding of the phenomenon under investigation. Quantitative and qualitative results in these articles were integrated and displayed using tables, figures, and diagrams to identify convergences and divergences, including personal interpretations. This interpretation of integrated findings from both arms helps us to have a more complete story or understanding of the phenomenon of interest. This would have not been possible with the use of isolated studies.

FINAL CONSIDERATIONS
MMAT was originally developed as a critical appraisal tool for systematic reviews of primary studies based on five different methodological designs, including mixed methods. In addition, it was considered in this study that nurse researchers can also use it as a tool to strive for methodological rigor when planning and conducting mixed methods studies. However, its use does not exempt the need for theoretical and methodological mastery of research with mixed methods by the interested scholars, in addition to compliance with the specific aspects of the quantitative and qualitative strands, which possibly transposes the tight use of the tool.