MIXED METHODS APPRAISAL TOOL: STRENGTHENING THE METHODOLOGICAL RIGOR OF MIXED METHODS RESEARCH STUDIES IN NURSING

MIXED METHODS APPRAISAL TOOL: FORTALECIMIENTE DEL RIGOR METODOLÓGICO DE INVESTIGACIONES DE MÉTODOS MIXTOS EN ENFERMERÍA

ABSTRACT

Objective:

to describe the use of the Mixed Methods Appraisal Tool to appraise and to strengthen the methodological rigor of mixed methods studies in nursing.

Method:

a theoretical essay was used to describe the application of Mixed Methods Appraisal Tool to support the development and assessment of mixed methods research in nursing. Four mixed methods articles in nursing were purposely chosen and evaluated based on the Mixed Methods Appraisal Tool criteria.

Results:

Mixed Methods Appraisal Tool is a tool for evaluating primary mixed methods studies based on five evaluation criteria: 1) justification for adopting mixed methods methodology; 2) integration between the quantitative and qualitative components; 3) interpretation of integrated findings of quantitative and qualitative data; 4) presentation of divergences between quantitative and qualitative results; and 5) compliance with the methodological rigor of each individual approach in mixed methods research.

Conclusion:

Mixed Methods Appraisal Tool is an instrumental resource that can be used to appraise mixed methods research and strengthen the methodological rigor in planning and conducting future mixed studies in nursing research.

DESCRIPTORS:
Research protocol; Nursing research; Methods; Evaluation studies as topic; Nursing

RESUMEN

Objetivo:

describir el uso de la Mixed Methods Appraisal Tool para evaluación y fortalecimiento del rigor metodológico de estudios de métodos mixtos en enfermería.

Método:

se utilizó un ensayo teórico para describir la aplicación de la Mixed Methods Appraisal Tool para apoyar el desarrollo y evaluación de la investigación de métodos mixtos en enfermería. Se eligieron y evaluaron intencionalmente cuatro artículos de enfermería de métodos mixtos con base en los criterios del instrumento.

Resultados:

la Mixed Methods Appraisal Tool es una herramienta para evaluar estudios primarios seleccionados a partir de revisiones de literatura mixta, que evalúa cinco modalidades / categorías metodológicas, incluida la investigación de métodos mixtos. En la evaluación de estudios vinculados al paradigma de investigación mixta, se emplean cinco criterios de evaluación: 1) justificación de la adopción de métodos mixtos; 2) integración entre los componentes cuantitativo y cualitativo; 3) interpretación de la integración entre datos cuantitativos y cualitativos; 4) presentación de divergencias entre resultados cuantitativos y cualitativos; y 5) cumplimiento del rigor metodológico de cada enfoque de investigación mixta, de modo particular.

Conclusión:

la Mixed Methods Appraisal Tool es un recurso instrumental que puede fortalecer el rigor metodológico en la planificación y realización de estudios mixtos de investigación en Enfermería.

DESCRIPTORES:
Protocolo de investigación; Investigación en enfermería; Métodos; Estudios de evaluación como asunto; Enfermería

RESUMO

Objetivo:

descrever o uso da Mixed Methods Appraisal Tool para avaliação e fortalecimento do rigor metodológico de estudos de métodos mistos em enfermagem.

Método:

foi utilizado um ensaio teórico para descrever a aplicação da Mixed Methods Appraisal Tool no desenvolvimento e avaliação da pesquisa de métodos mistos em enfermagem. Quatro artigos de métodos mistos em enfermagem foram intencionalmente escolhidos e avaliados com base nos critérios do instrumento.

Resultados:

a Mixed Methods Appraisal Tool é uma ferramenta para avaliar estudos de métodos mistos primários com base em cinco critérios de avaliação: 1) justificativa para a adoção da metodologia de métodos mistos; 2) integração entre os componentes quantitativos e qualitativos; 3) interpretação de resultados integrados de dados quantitativos e qualitativos; 4) apresentação de divergências entre resultados quantitativos e qualitativos; e 5) conformidade com o rigor metodológico de cada abordagem individual na pesquisa de métodos mistos.

Conclusão:

a Mixed Methods Appraisal Tool é um recurso instrumental que pode ser usado para avaliar a pesquisa de métodos mistos e fortalecer o rigor metodológico no planejamento e na condução de futuros estudos mistos na pesquisa em enfermagem.

DESCRITORES:
Protocolo de pesquisa; Pesquisa em enfermagem; Métodos; Estudos de avaliação como assunto; Enfermagem

INTRODUCTION

To strengthen the methodological rigor of scientific research and robust publications, guidelines and protocols for the planning, execution and evaluation of research studies have been increasingly developed and used by various disciplines, including health and nursing. Examples of guidelines include the International Committee of Medical Journal Editors (ICMJE)1 and Enhancing the Quality and Transparency of Health Research (EQUATOR network)2. Due to the insufficient reporting of study findings in the literature, the international EQUATOR network was established 15 years ago with the aim of promoting higher quality and more transparent reporting of different study designs2. To date, the network comprises 463 reporting guidelines in the form of checklists, including guidelines for randomized clinical trials3, observational studies (cohort, case-control and cross-sectional)4, systematic reviews and meta-analyses5, qualitative studies6 and other study types2^,7.

These guidelines can help increase robust and transparent reporting on the production of scientific knowledge, which is of critical interest to researchers/authors, editors and reviewers of scientific journals. In general, the guidelines require reporting with clarity and accuracy for the title, research problem, state of the science, study objective(s) and design, method and so on3^-7.

In the areas of health and nursing, the use of recommendations for the production of research reports is grounded in the context of evidence-based practice based on scientific and rigorous work that leads to knowledge translation by professionals. Yet, a recent literature review of research studies from seven countries revealed that nurses in the hospital context still have insufficient knowledge to critically appraise scientific evidence and difficulties in implementing evidence-based practice8.

In some parts of the world, there is a tendency for nurses to favor studies that are grounded in positivism - a fact that can be explained by the strong influence of the biomedical model in the profession8^-9. However, it is known that the plurality of scientific research in nursing is real. Therefore, nurse researchers may use diverse theoretical-methodological approaches based on the study objectives to generate new knowledge10.

Within the range of possibilities for scientific research in nursing lies the field of mixed methods research, which is made up of both quantitative and qualitative designs11. In this sense, mixed methods research has emerged as a new scientific paradigm which helps us to deepen our understanding about complex health issues for the betterment of human beings10^,12. We recommend that, if reasonable and possible, nurse researchers opt for mixed methods research when examining and intervening in complex social and health problems. Mixed methods research, even though can be laborious and challenging, its findings can promote a richer quality of life and better care for our clients, lead to the increased visibility of the nursing discipline, and advance nursing science overall10^,13.

Although there has been an increase in the use of mixed methods methodology in nursing, the research process including data integration and findings dissemination are, in general, still embryonic among nurse researchers in many parts of the world9^,14. Evidence shows that some mixed methods studies in nursing seem to have methodological weaknesses, including a lack of clarity and insufficient information as to the purpose; research design/justification; weights assigned to the quantitative and qualitative arms and data integration strategies9^,14.

Therefore, it is considered that the correct application of the mixed nursing research methodology is a challenge for researchers in the field. Thus, there is a need to increase means/strategies/resources that enable actions for the rigor and scientific soundness of research with this innovative methodological paradigm. One possibility for this is the adoption of the Mixed Methods Appraisal Tool (MMAT), a tool that outlines the object of this study and that can be useful in strengthening the methodological rigor of mixed methods studies15.

This study aimed to describe the use of the Mixed Methods Appraisal Tool to appraise and to strengthen the methodological rigor of mixed methods studies in nursing.

METHOD

This is a theoretical-reflective essay, which involved problematizing the theme, reading and theoretical abstraction of technical-scientific works on mixed methods of research. It is noteworthy that the team of authors of the study was composed of researchers with experience in the use of mixed methods in Nursing research.

This study involves a description of MMAT in three areas: 1) Mixed Methods Appraisal Tool: 1) origin and characteristics; 2) appraisal criteria; and 3) its application to critically appraise the methodological rigor. For the development of the item 3, four mixed research studies developed in Nursing were selected intentionally to exemplify the use of the tool. For this selection, methodological rigor, originality and topicality were considered.

MIXED METHODS APPRAISAL TOOL: ORIGIN AND CHARACTERISTICS

MMAT is an assessment tool for scientific studies developed in 200616 by researchers from Canada (McGill University, Department of Family Medicine) and the United Kingdom, linked to colleges/departments in the health area, including Nursing. It was reformulated17 and updated15 after a literature review and critical analysis carried out by experts, using the Delphi technique18. The previous version of MMAT dates back to 2011 and is translated and cross-culturally adapted to the Brazilian reality19. However, the most current option of the tool is from 201815 and is available in English and French. All versions have unrestricted open access.

Originally, MMAT was developed to assess the quality of primary studies recruited from mixed literature review research studies15^-17, since it includes research evaluation criteria from five methodological categories: randomized clinical trials; non-randomized clinical trials; observational quantitative studies (cross-sectional, case-control and cohort), qualitative studies, and mixed-methods research15. It is therefore assumed that MMAT can be a useful tool for investigative planning, in addition to its evident purpose in evaluating primary studies.

Although MMAT was developed with the main purpose of helping to conduct systematic reviews15^,17^,19, the authors encourage the use of additional assessments, such as those already mentioned according to the EQUATOR network.2. In addition, it is recommended that the evaluation is performed by two researchers/reviewers, independently, and that they have experience in the development of the study designs to be appreciated15.

In its structuring, in addition to introductory/conceptual elements, an explanatory diagram for defining the type of study to be evaluated and the technical-scientific basis of the tool, MMAT15^,17 has two parts. The first consists of the evaluation form itself, presented as a checklist. In this part, there are six “blocks”, of which the first refers to “filter questions”, which are applied to the the five (all) methodological categories of the studies that can be appreciated by the tool. In turn, the other five components are specifically related to study designs to be evaluated, in the following order: qualitative studies, randomized clinical trials, non-randomized clinical trials, descriptive quantitative studies (observational epidemiological - cross-sectional, cohort and case-control), and mixed-methods research15^,17.

In the current version, for each methodological category of studies evaluated by MMAT, five check items are applied, which determine compliance with the established evaluation criteria. These items have options for answering yes, no or indeterminate, in addition to a space for comment(s) by the evaluators15.

MMAT as a whole has 25 specific evaluation items/criteria (five from each methodological modality), in addition to two previous questions that are common to all types of study, that is, each study is subjected to the evaluation of seven items/criteria15. In the practice, this means that each of the five methodological categories that can be evaluated (that is, each study) is subjected to five evaluation criteria specific to this methodological category, in addition to two questions applicable to any of these categories. In addition to that, it is important to emphasize that the authors of MMAT suggest that, in case of a negative or indefinite answer to one or to the first two general questions, it is to be considered that the research is not eligible or susceptible to be appreciated by the tool15^,17.

The general items contained in MMAT concern the clear existence of definition of the research question(s) and/or objective(s), as well as whether the data presented by the study are sufficient to answer these questions15^,17. It is interpreted that this may mean that, by not stating a research purpose or by enunciating it vaguely/inconsistently, in addition to not having information applicable to the answers to a certain purpose, the study should not be evaluated, because it does not have enough empirical information to confer it scientific research status.

The specific items/questions to the evaluation of each methodological category in MMAT are naturally linked to the means that boost or ensure the scientific-methodological rigor of each study design15. Despite recognizing the importance and the potential contribution of each of the evaluation possibilities by the tool, it is prudent to reassert that the central object of this study is to debate MMAT in the specialized context of mixed-methods research studies, envisioning potentialities in addition to being an evaluative tool, but also a potential contribution to the rigorous planning of mixed studies, especially in the area of Nursing.

After duly evaluated, each study submitted for consideration by MMAT can obtain a more quality complete and accurate assessment of its methodological. This is illustrated in a systematic mixed-methods review that assigns proportions of conformity to the quality to the studies evaluated20, according to the 2011 version of MMAT17.

It is worth noting that the current version of MMAT15 incorporated two questions into the specific evaluation block for research using mixed methods compared to the previous version, which has three questions/items for this study design17. It is believed that this signal increases in the criteria for defining the methodological quality in this recent scientific paradigm, which is important and necessary, since difficulties and even inconsistencies in its proper conduction are recognized10^-11^,14. Thus, the knowledge of criteria that assist in the methodological rigor of mixed-methods research is of interest for the Nursing science.

MMAT CRITERIA FOR EVALUATING MIXED METHODS RESEARCH STUDIES

For the purpose of the present article, we focus solely on five specific criteria to appraise the methodological rigor of mixed methods studies at MMAT: 1) “Is there an adequate rationale for using a mixed methods design to address the research question?” (justification); 2) “Are the different components of the study effectively integrated to answer the research question?” (integration); 3) “Are the outputs of the integration of qualitative and quantitative components adequately interpreted?” (interpretation); 4) “Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?” (disagreements); and 5) “Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?” (adherence)15. The response options of “yes”, “no” or “can’t tell” are the same as those of the two screening questions15. It is worth noting that the current version of MMAT15 has two more questions/criteria than the 2011 version, an effort to increase the methodological rigor for mixed methods research appraisal as a way to recognize and address methodological and reporting inconsistencies and inadequacy10^-11^,14.

MMAT Criterion 1: Justification - “Is there an adequate rationale for using a mixed methods design to address the research question?”15.

When appraising a mixed methods study based on this criterion, the reviewer should look for the rationale for the needs for both quantitative and qualitative arms. The rationale should be clear and justifiable based on the state of the science and gap identifications. In general, there are two main reasons for using mixed methods research: complementarity and sequentiality11. Complementarity is used to provide a more complete story of the investigated phenomenon. In other words, quantitative data are complementary to qualitative data, or vice versa; with only one type of data, the phenomenon cannot be fully understood21. For sequentially, one research approach comes before the other approach based on the research questions and the needs to fill the gaps for the phenomenon of interest. The qualitative arm can come before or after the quantitative arm, or vice versa, again, based on the state of the science and the gap identification11.

MMAT Criterion 2: Integration - “Are the different components of the study effectively integrated to answer the research question?”15.

In mixed methods research, a basic premise is the need for a clear integration between quantitative and qualitative results, as opposed to presenting research results in each arm separately. 13. Data integration, a crucial aspect in mixed methods research, is defined as a clear presentation of the association between the quantitative and qualitative arms12^,15. The decision on how integration should take place depends on the study objectives, design, data collection and analysis strategies, as well as the researchers’ experiences and skills. In sequential studies, for example, the integration may occur during the data analysis in the first phase that guides the data collection process in the second phase22. Data can also be merged especially in convergent mixed methods studies11. Creswell23 classified integration types in mixed methods into four categories: data merging, data explanation, data building, and data embedding:

• Data merging: qualitative and quantitative data are analyzed separately, then findings from both arms are compared for differences and similarities in a convergent design24.

• Data explanation: qualitative findings help explain quantitative findings in an explanatory sequential design22.

• Data building: qualitative findings inform future quantitative research questions and design in an exploratory sequential design.

• Data embedding: qualitative findings can be embedded and help to explain quantitative findings in an intervention design21.

Strategies used to display data integration may include a joint-display matrix and a visual representation of quantitative and qualitative findings. Such strategies can facilitate clear presentations of findings in both arms and are more visually attractive, displaying how the findings are compared, connected, embedded, and interrelated. In some cases, data conversion is used-transforming qualitative data into quantitative numerical data, or vice versa11^,22.

Recently, authors25 from the United Kingdom proposed a step-by-step model for data integration in mixed methods research, the Pillar Integration Process (PIP). PIP comprises four stages: listing, correspondence, verification, and construction with the end product as a joint display that clearly represents the integration of qualitative and quantitative findings25.

MMAT Criterion 3: Interpretation - “Are the outputs of the integration of qualitative and quantitative components adequately interpreted?”15.

After the integration of qualitative and quantitative results, the findings need to be interpreted based on the similarities and divergences (disagreements, discrepancies or dissonances). A successful integration should result in a view closer to the integral of the phenomenon of interest, rather than the sum of the parts (findings) from each arm in isolation11^,23^-24.

MMAT Criterion 4. Disagreements - “Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?”15.

For this criterion, when there are no disagreements between quantitative and qualitative results, the reviewer rates “Yes”. However, if disagreements arise, to rate this criterion as “Yes”, a clear explanation is needed as to how such disagreements are handled and interpreted15.

MMAT Criterion 5. Adherence - Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?”15.

This criterion involves an appraisal of the research method on the quantitative and qualitative arms, separately. To evaluate the quantitative arm, use MMAT Part I: study types 2 (randomized controlled trails), 3 (quantitative non-randomized) or 4 (quantitative descriptive), accordingly and use study type 1 for the qualitative arm15. Both arms need to meet completely the criteria of scientific rigor to be considered of good quality for the mixed methods study15. When both arms are rated different for the level of quality, the overall quality of the mixed methods study must be assigned to the lower rating15. For instance, if the quantitative arm is rated with “high” quality while the qualitative arm is rated as “low” quality, the overall quality of the mixed methods study must be “low”.

In sum, MMAT can be used to appraise a mixed methods study using two screening questions and additional questions based on the research questions and the types of the quantitative and qualitative arms. The integration in a mixed methods study must take a holistic approach, occurring at some point during the research process: data collection, data analysis, and when findings emerged22. Such an integration should result in meta-inference, generating a “complete” story of the phenomenon of interest22^-23. Instead of conducting two or more separate studies on one topic, applying mixed methods research to generate meta-inferences can be valuable, critical for nurses whose clients’ health and life circumstance are usually complex which also demand complex answers10^,15.

APPLICATION OF MMAT TO APPRAISE METHODOLOGICAL RIGOR OF MIXED METHOD STUDIES IN NURSING

Across the globe, even though researchers in different disciplines have increasingly acknowledged the benefits of mixed methods research, some nurse researchers still subscribe to either quantitative or qualitative research methodology because their methodology of choice is aligned with their perspective/vision into the research problem10.

Planning and conducting mixed methods research can be laborious and challenging because both qualitative and quantitative arms must be well planned, executed, and integrated. If one arm’s methodological rigor falls off, the other arm and the overall study may suffer. In this section, we purposefully selected four mixed methods studies based on their originality and common topical area on nurses’ perceptions in different fields: one from Australia, two from Brazil, and one from Denmark26^-29. We demonstrate the use of MMAT15 in a table format to appraise the selected mixed methods studies. Due to space limitations, we only appraise mixed methods criteria (1-4), omitting criterion 5 that requires an appraisal of the methodological rigor in the qualitative and quantitative arms separately15.

Study 1: researchers in Australia used an exploratory sequential mixed methods design to explore and describe nurses' attitudes and behavior towards patients’ use of complementary therapies. They collected qualitative data in the first phase from 19 professionals through semi-structured interviews. Based on the themes emerging from this stage, a quantitative research instrument was developed and administered online to 614 nurses across the country. For data integration, data building was used in which the results of the qualitative interview were applied to establish the questions asked in a national survey26.

Study 2: researchers in southern Brazil used an explanatory sequential mixed methods design to examine the association between accreditation and nursing team’s satisfaction. In the first phase (quantitative correlational cross-sectional), the Brazilian version of the Index of Work Satisfaction was administered to 226 nursing staff at three general hospitals (one private/certified in excellence by the National Accreditation; one private/non-certified; and one public/non-certified). Results showed that nurses from the certified hospital were more satisfied with their work than those from non-certified hospitals. The second phase (qualitative) complemented the quantitative phase through interviews (n=39). Data were analyzed by Collective Subject Discourse (CSD) technique. Through the use of a joint display, the CSD findings were presented in a table format with the quantitative results, which were enriched and supported by the qualitative findings27.

Study 3: researchers in Brazil explored healthcare professionals’ resilience levels and processes in caring for people with mental disorders in a psychiatric hospital. A convergent mixed methods design was used. In the quantitative cross-sectional arm, the Resilience Scale was administered to 40 health professionals, most of whom were nursing technicians, followed by physicians, and registered nurses. In the qualitative arm, Grounded Theory was applied (n=26) using initial and focused coding. Quantitative data showed that nursing clinicians had the lowest level of resilience compared to other groups. Younger healthcare professionals with fewer years of work experience had higher scores than those who were older with more years of work experience. Qualitative data enriched resilience quantitative findings in this study28.

Study 4: researchers in Denmark used a convergent mixed methods design to examine the effect of the "Rehabilitation 24/7" education intervention on nursing staff’s ability, opportunity and motivation to work in a rehabilitation setting. Quantitative data were collected pre- and post-education (n=40). Qualitative data (n=10) were analyzed by thematic analysis, guided by an existing model of ability (physical and psychological), opportunity (physical and social) and motivation (automatic and reflective). Data integration was carried out during the discussion process. The authors concluded that based on qualitative findings, the educational intervention strengthened the nursing staff’s knowledge and beliefs about patient rehabilitation, in addition to increasing awareness of their own role29.

Chart 1 summarizes the fulfillment of four MMAT criteria by the aforementioned examples, separately.

Chart 1 -
Examples of compliance with the evaluation items/criteria for the MMAT's mixed-methods research in studies in the Nursing area. Brazil, 2020.

Criteria	Study 126	Study 227	Study 328	Study 429
1) Justification	Application of qualitative arm for themes generation and quantitative arm for instrument development to examine the phenomenon of interest.	Gaining a deeper understanding of the phenomenon investigated.	Identifications of convergences and differences between qualitative and quantitative results.	Use of both methods combined to gain a better understanding about the education program.
2) Integration	A quantitative questionnaire was developed based on the qualitative results in phase 1.	Results from quantitative arm were compared and contrasted to qualitative results. A joint display was used to present the integration of the final results.	A diagram presented to compare between quantitative and qualitative results.	Data were integrated in the discussion section as a diagram, comprising the methodological path and final findings.
3) Interpretation	To justify the evidence using triangulated qualitative and quantitative data to improve the findings.	Qualitative findings supported quantitative findings about staff’s satisfaction working for an accredited hospital.	Younger healthcare professionals with fewer years of work experience had higher scores than those who were older with more years of work experience. Qualitative data enriched resilience quantitative findings.	Quantitative results showed no major improvements after the education program. Qualitative results helped enrich the understanding of the changes in the nursing team regarding the members' behaviors after the intervention.
4) Disagreements	Results of the quantitative survey on the “Integrating complementary therapy” item were somewhat consistent with results in the qualitative arm.	Results in both quantitative and qualitative arms showed similarities in that nurses working for an accredited hospital were more satisfied with their work than those working for non-accredited hospitals.	Factor III in the resilience scale had the lowest score when compared to other factors. Qualitative results showed that healthcare professionals constantly searched for positive adaptations to the work situations.	The quantitative results did not show any significant change after the intervention. However, qualitative results revealed immediate unit structural changes after the educational program.

In sum, the appraisals of the selected articles presented in Chart 1 show that mixed methods research can enrich the understanding of the phenomenon under investigation. Quantitative and qualitative results in these articles were integrated and displayed using tables, figures, and diagrams to identify convergences and divergences, including personal interpretations. This interpretation of integrated findings from both arms helps us to have a more complete story or understanding of the phenomenon of interest. This would have not been possible with the use of isolated studies.

FINAL CONSIDERATIONS

MMAT was originally developed as a critical appraisal tool for systematic reviews of primary studies based on five different methodological designs, including mixed methods. In addition, it was considered in this study that nurse researchers can also use it as a tool to strive for methodological rigor when planning and conducting mixed methods studies.However, its use does not exempt the need for theoretical and methodological mastery of research with mixed methods by the interested scholars, in addition to compliance with the specific aspects of the quantitative and qualitative strands, which possibly transposes the tight use of the tool.

REFERENCES

• 1. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals [Internet]. 2019 [cited 2020 June 19]. Available from: http://www.icmje.org/icmje-recommendations.pdf
  » http://www.icmje.org/icmje-recommendations.pdf
• 2. Enhancing the Quality and Transparency of Health Research - Equator Network [Internet]. Oxford (UK): Equator Network; 2020 [cited 2020 June 19]. Available from: https://www.equator-network.org/
  » https://www.equator-network.org/
• 3. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med [Internet]. 2010 [cited 2019 Nov 02];152(11):726-32. Available from: https://doi.org/10.7326/0003-4819-152-11-201006010-00232
  » https://doi.org/10.7326/0003-4819-152-11-201006010-00232
• 4. Von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vanderbroucke JP, STROBE Initiative. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet [Internet]. 2007 [cited 2019 Nov 02];370(9596):1453-7. Available from: https://doi.org/10.1016/S0140-6736(07)61602-X
  » https://doi.org/10.1016/S0140-6736(07)61602-X
• 5. Moher D, Liberati A, Tetzlaff J, Altman DG , PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg [Internet]. 2010 [cited 2019 Nov 02];8(5):336-41. Available from: https://doi.org/10.1016/j.ijsu.2010.02.007
  » https://doi.org/10.1016/j.ijsu.2010.02.007
• 6. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care [Internet]. 2007 [cited 2019 Nov 02];19(6):349-57. Available from: https://doi.org/10.1093/intqhc/mzm042
  » https://doi.org/10.1093/intqhc/mzm042
• 7. Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for Quality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. J Nurs Care Qual [Internet]. 2016 [cited 2019 Nov 02];31(1):1-8. Available from: https://doi.org/10.1097/NCQ.0000000000000153
  » https://doi.org/10.1097/NCQ.0000000000000153
• 8. Camargo FC, Iwamoto HH, Galvão CM, Pereira GA, Andrade RB, Masso GC. Competences and barriers for the evidence-based practice in nursing: an integrative review. Rev Bras Enferm [Internet]. 2018 [cited 2020 Nov 02];71(4):2030-8. Available from: https://doi.org/10.1590/0034-7167-2016-0617
  » https://doi.org/10.1590/0034-7167-2016-0617
• 9. Irvine FE, Clark MT, Efstathiou N, Herber OR, Howroyd F, Gratrix L, et al. The state of mixed methods research in nursing: A focused mapping review and synthesis. J Adv Nurs [Internet]. 2020 [cited 2020 Nov 02];76:2798-809. Available from: https://doi.org/10.1111/jan.14479
  » https://doi.org/10.1111/jan.14479
• 10. Oliveira JLC, Magalhães AMM, Matsuda LM. Mixed method in nursing research: application possibilities according to Creswell. Texto Contexto Enferm [Internet]. 2018 [cited 2020 Nov 02];27(2):e0560017. Available from: https://doi.org/10.1590/0104-070720180000560017
  » https://doi.org/10.1590/0104-070720180000560017
• 11. Creswell JW, Clarck VLP. Designing and conducting mixed methods research. Thousand Oaks, CA(US): Sage; 2017 [cited 2020 Nov 02]. Available from: https://www.bookdepository.com/designing-conducting-mixed-methods-research-john-w-creswell/9781483344379
  » https://www.bookdepository.com/designing-conducting-mixed-methods-research-john-w-creswell/9781483344379
• 12. Oliveira JLC. Data integration in mixed-method research studies: challenge and opportunity for nursing. Texto Contexto Enferm [Internet]. 2020 [cited 2020 Nov 02];29:e20200203. Available from: https://doi.org/10.1590/1980-265X-TCE-2020-0002-0003
  » https://doi.org/10.1590/1980-265X-TCE-2020-0002-0003
• 13. Viera CS, Bugs BM, Carvalho ARS, Gaiva MAM, Toso BRGO. Description of the use of integrative mixed method in neonatal nursing. Rev Esc Enferm USP [Internet]. 2019 [cited 2020 Nov 02];53:e03408. Available from: https://doi.org/10.1590/s1980-220x2017039303408
  » https://doi.org/10.1590/s1980-220x2017039303408
• 14. Younas A, Pedersen M, Tayaben JL. Review of mixed-methods research in nursing. Nurs Res [Internet]. 2019 [cited 2020 Nov 02];68(6):464-72. Available from: https://doi.org/10.1097/NNR.0000000000000372
  » https://doi.org/10.1097/NNR.0000000000000372
• 15. Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods appraisal tool (MMAT) Version 2018: user guide [Internet]. Montreal (CA): University McGill; 2018 [cited 2020 June 19]. Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/mmat_2018_criteria-manual_2018-08-01_eng.pdf
  » http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/mmat_2018_criteria-manual_2018-08-01_eng.pdf
• 16. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A Scoring system for appraising Mixed Methods research, and concomitantly appraising qualitative, quantitative and Mixed Methods primary studies in Mixed Studies reviews. Int J Nurs Stud [Internet]. 2009 [cited 2019 Nov 02];46(4):529-46. Available from: https://doi.org/10.1016/j.ijnurstu.2009.01.009
  » https://doi.org/10.1016/j.ijnurstu.2009.01.009
• 17. Mixed Methods Appraisal Tool (MMAT) - Version 2011 [Internet]. Montreal(CA): University McGill; 2011 [cited 2020 June 19]. Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/84371689/MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.08.21.pdf
  » http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/84371689/MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.08.21.pdf
• 18. Hong QN, Pluye P, Fàbreges S, Bartlett G, Boardman F, Cargo M, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. J Clin Epidemiol [Internet]. 2019 [cited 2019 Nov 02];111:49-59.e1. Available from: https://doi.org/10.1016/j.jclinepi.2019.03.008
  » https://doi.org/10.1016/j.jclinepi.2019.03.008
• 19. Souto RQ, Lima KSA, Pluye P, Hong QN, Djogovic T, Araújo GKN. Translation and cross-cultural adaptation of the mixed methods appraisal tool the brazilian context. Rev Pesq Cuid Fundam Online [Internet]. 2020 [cited 2020 Nov 02]; 2:510-6. Available from: https://doi.org/10.9789/2175-5361.rpcfo.v12.8615
  » https://doi.org/10.9789/2175-5361.rpcfo.v12.8615
• 20. Salviano CF, Gomes PL, Martins G. Lived experiences by families and children with urinary and intestinal symptoms: systematic review of mixed methods. Esc Anna Nery [Internet]. 2020 [cited 2020 Nov 02];24(3):e20190137. Available from: https://doi.org/10.1590/2177-9465-EAN-2019-0137
  » https://doi.org/10.1590/2177-9465-EAN-2019-0137
• 21. Ross R, Sawatphanit W, Suwansujarid T, Stidham AW, Drew D, Creswell JW. The effect of telephone support on depressive symptoms among HIV-positive pregnant women in Thailand: An embedded mixed-methods study. J Assoc Nurses AIDS Care [Internet]. 2013 [cited 2020 Nov 02];24(5):e13-24. Available from: https://doi.org/10.1016/j.jana.2012.08.005
  » https://doi.org/10.1016/j.jana.2012.08.005
• 22. Boongird C, Ross R. Views and expectations of Thai community-dwelling elderly in reporting falls to their primary care physicians: an explanatory sequential mixed methods study. J Appl Gerontol [Internet]. 2017 [cited 2020 Nov 02];36(4):480-98. Available from: https://doi.org/10.1177/0733464815606799
  » https://doi.org/10.1177/0733464815606799
• 23. Creswell JW. A concise introduction to mixed methods research. Thousand Oaks, CA(US): Sage; 2015.
• 24. Santos JLG, Erdmann AL, Meirelles BHS, Lanzoni GMM, Cunha VP, Ross R. Integração entre dados quantitativos e qualitativos em uma pesquisa de métodos mistos. Texto Contexto Enferm [Internet]. 2017 [cited 2018 Feb 01]; 26(3):e1590016. Available from: https://doi.org/10.1590/0104-07072017001590016
  » https://doi.org/10.1590/0104-07072017001590016
• 25. Johnson RE, Grove AL, Clarke A. Pillar integration process: a joint display technique to integrate data in mixed methods research. J Mix Methods Res [Internet]. 2017 [cited 2019 Nov 02];13(3):301-20. Available from: https://doi.org/10.1177/1558689817743108
  » https://doi.org/10.1177/1558689817743108
• 26. Hall H, Brosnan C, Cant R, Collins M, Leach M. Nurses’ attitudes and behaviour towards patients’ use of complementary therapies: a mixed methods study. J Adv Nurs [Internet]. 2018 [cited 2020 Nov 02];74(7):1649-58. Available from: https://doi.org/10.1111/jan.13554
  » https://doi.org/10.1111/jan.13554
• 27. Oliveira JLC, Magalhães AMM, Bernardes A, Haddad MCFL, Wolff LDG, Marcon SS, et al. Influence of hospital accreditation on professional satisfaction of the nursing team: mixed method study. Rev Latino-Am Enfermagem [Internet]. 2019 [cited 2020 Nov 02];27:e3109. Available from: https://doi.org/10.1590/1518-8345.2799.3109.
  » https://doi.org/10.1590/1518-8345.2799.3109.
• 28. Brolese DF, Lessa G, Santos JLG, Mendes JS, Cunha KS, Rodrigues J. Resilience of health team in caring for people with mental disorders in a psychiatric hospital. Rev Esc Enferm USP [Internet]. 2017 [cited 2019 Nov 02];51:e03230. Available from: https://doi.org/10.1590/s1980-220x2016026003230
  » https://doi.org/10.1590/s1980-220x2016026003230
• 29. Loft MI, Esbensen BA, Kirk K, Pedersen L, Martinsen B, Iversen H, et al. Nursing staffs self-perceived outcome from a rehabilitation 24/7 educational programme - a mixed-methods study in stroke care. BMC Nursing [Internet]. 2018 [cited 2019 Nov 02];17:17. Available from: https://doi.org/10.1186/s12912-018-0285-z
  » https://doi.org/10.1186/s12912-018-0285-z

Edited by

EDITORS

Associated Editors: Mara Ambrosina de Oliveira Vargas, Gisele Cristina Manfrini, Ana Izabel Jatobá de Souza. Editor-in-chief: Roberta Costa.

Publication Dates

• Publication in this collection
  27 Aug 2021
• Date of issue
  2021

History

• Received
  13 Dec 2020
• Accepted
  01 Apr 2021