Employing Kirkpatrick’s framework to evaluate nurse training: an integrative review^*

Objective to evaluate the evidence on the use of Donald Kirkpatrick’s framework in nursing training evaluation.

Method integrative literature review in the Latin American and Caribbean Health Sciences Literature, Medical Literature Analysis and Retrieval System and Web of Science databases. Studies that answered the review question “Which is the evidence in using Donald Kirkpatrick’s framework to evaluate training in the nursing workplace?” published in Portuguese, English, or Spanish were included.

Results out of 108 studies retrieved, thirteen were included. The majority evaluated the four levels proposed in the model (reaction, learning, behavior, and results) or, at least, a combination of the first three ones. Different instruments were used to evaluate nursing training, mainly in quantitative approaches for reaction and learning levels and qualitative for behavior and results levels. This approach highlights the flexibility of the model and the importance of choosing a reliable set of instruments, which is crucial to qualify the analysis at each level.

Conclusion Kirkpatrick’s model has been used worldwide to evaluate training in the nursing field and has been shown to be suitable for it, as long as there is an appropriate selection of instruments at each level.

Descriptors
Continuing Education; Nurses; Evaluation of the Efficacy-Effectiveness of Interventions; Review; Methods; Inservice Training

Highlights:

(1) Kirkpatrick’s framework is effective for evaluating various nursing training.

(2) The framework upholds the choice of measuring instruments for each level.

(3) The four-levels or a combination of the first three were the most commonly used to evaluate training.

(4) The four-levels or a combination of the first three were the most common to evaluate training.

(5) Evaluation of results in organizational practices is the most challenging level.

Objetivo: evaluar la evidencia sobre el uso del modelo de Donald Kirkpatrick en la evaluación de la formación en enfermería

Método: revisión bibliográfica integradora en las bases de datos Literatura Latinoamericana y del Caribe en Ciencias de la Salud, Medical Literature Analysis and Retrieval System y Web of Science. Se incluyeron los estudios que respondían a la pregunta de revisión “¿Cuál es la evidencia para el uso del modelo de Donald Kirkpatrick para evaluar la formación en el trabajo de enfermería?” publicados en portugués, inglés o español.

Resultados: de los 108 estudios recuperados, se incluyeron trece. La mayoría evaluaba los cuatro niveles propuestos en el modelo (reacción, aprendizaje, comportamiento y resultados) o al menos una combinación de los tres primeros. Se utilizaron diferentes instrumentos para evaluar la formación en enfermería, principalmente enfoques cuantitativos para los niveles de reacción y aprendizaje y enfoques cualitativos para los niveles de comportamiento y resultados. Este enfoque pone de manifiesto la flexibilidad del modelo y la importancia de elegir un conjunto fiable de instrumentos, lo cual es fundamental para cualificar el análisis en cada nivel.

Conclusión: el modelo de Kirkpatrick se ha utilizado en todo el mundo para evaluar la formación en enfermería y ha demostrado ser adecuado para este fin, siempre que haya una selección adecuada de instrumentos en cada nivel.

Descriptores
Educación Continua; Enfermeras y Enfermeros; Evaluación de Eficacia-Efectividad de Intervenciones; Revisión; Métodos; Capacitación en Servicio

Destacados:

(1) La metodología de Kirkpatrick es eficaz para evaluar los distintos niveles de formación de los profesionales de enfermería.

(2) La metodología apoya la elección de instrumentos de medida para cada nivel.

(3) Los cuatro niveles o una combinación de los tres primeros fueron los más utilizados para evaluar la formación.

(4) Los cuatro niveles o una combinación de los tres primeros fueron los más frequentes para evaluar la formación.

(5) La evaluación de los resultados en las prácticas organizativas es el nivel más desafiante.

Objetivo: avaliar as evidências sobre o uso do modelo de Donald Kirkpatrick na avaliação da formação em enfermagem.

Método: revisão integrativa da literatura nas bases de dados Literatura Latino-Americana e do Caribe em Ciências da Saúde, Medical Literature Analysis and Retrieval System e Web of Science. Foram incluídos estudos que respondem à questão de revisão “Quais são as evidências da utilização do modelo de Donald Kirkpatrick para avaliar o treinamento no ambiente de trabalho da enfermagem?” publicados em português, inglês ou espanhol.

Resultados: dos 108 estudos recuperados, treze foram incluídos. A maioria avaliou os quatro níveis propostos no modelo (reação, aprendizagem, comportamento e resultados) ou, pelo menos, uma combinação dos três primeiros. Foram utilizados diferentes instrumentos para avaliar a formação em enfermagem, principalmente em abordagens quantitativas para os níveis de reação e aprendizagem e qualitativas para os níveis de comportamento e resultados. Esta abordagem evidencia a flexibilidade do modelo e a importância da escolha de um conjunto confiável de instrumentos, fundamental para qualificar a análise em cada nível.

Conclusão: o modelo de Kirkpatrick tem sido utilizado em todo o mundo para avaliar a formação na área da enfermagem e tem se revelado adequado para o objetivo, desde que haja uma seleção adequada de instrumentos em cada nível.

Descritores
Educação Continuada; Enfermeiras e Enfermeiros; Avaliação de Eficácia-Efetividade de Intervenções; Revisão; Método; Capacitação em Serviço

Destaques:

(1) A metodologia de Kirkpatrick é eficaz na avaliação de vários níveis de formação em enfermagem.

(2) A metodologia suporta a escolha de instrumentos de medida para cada nível.

(3) Os quatro níveis ou uma combinação dos três primeiros foram os mais utilizados para avaliar a formação.

(4) Os quatro níveis ou uma combinação dos três primeiros foram os mais comuns para avaliar a formação.

(5) A avaliação dos resultados nas práticas organizacionais é o nível mais desafiante.

Introduction

Training healthcare personnel has been incorporated as a strategy for talent retention in human resources management. Training and development practices at the workplace are described as dynamic and continuous processes aimed at promoting social advances, increasing resolution and fostering better health outcomes(1).

Training can be an opportunity for professional development and increased productivity, creativity, and innovation. These are the main aspects of high-quality services and teams effectiveness(2). Lack of training can negatively influence perception of stress, increase turnover and stress, and reduce work performance(3).

Training strategies are key to improve health systems in low-income countries(4). The development of workers through training is a critical indicator of effective management(5). In Brazil, training is also political, since knowledge interacts as an essential part of the power struggle and meaningful learning could lead to deep-rooted changes in healthcare(6).

Evaluating training programs can help managers to understand the effectiveness and sustainability of the learning process to produce changes in the short or long term. However, selecting a suitable framework for evaluation is not an easy task and many interventions struggle with low or very low levels of evidence for targeted outcomes(4).

In 1959, Donald Kirkpatrick published a framework to support managers to evaluate the results of training and development practices among workers and organizational systems(7-8), based on summative assessments. The model is composed of four levels of evaluation: Reaction, Learning, Behavior and Results(7). Reaction (level I) encompasses the participant’s perceptions of their learning experience, program structure, content, teaching methods, and instructional aspects, such as materials and the quality of instruction. Learning (level II) is related to changes in attitudes, perceptions, knowledge and skills. Behavior (level III) evaluates the knowledge transfer to the workplace through behavioral changes. Finally, the Results level (level IV) evaluates the applicability of intervention results to change the organizational practice and improve the outcomes(7-8).

Managers and researchers should carefully consider adopting an evaluation system based on this framework, as the implementation of isolated levels is not recommended and higher level of evaluation do not lead to more valuable results if they are isolated(7).

The Kirkpatrick model has been widely accepted in the scientific community to support development and training in several areas, including health(9-14). Several improvements were described by the Kirkpatrick Group since the first publication, arriving at a new model called the New World Kirkpatrick model in 2016(15). Despite the improvements, there are criticisms regarding the applicability of the Kirkpatrick model(16-18), the major concern being the difficulty in implementing all four levels(16). These criticisms prompted this review to ensure the method’s applicability in nursing. The aim of this study is to evaluate the evidence on the use of Donald Kirkpatrick’s framework in the evaluation of nursing training.

Methods

Type of study

We used an integrative literature review, based on the Whittemore and Knafl framework, carried out in five stages: problem identification; literature search/screening; data evaluation; data analysis; presentation of the review(19). We choose this methodological structure to ensure a systematic approach to evidence synthesis. The intended outcome is to obtain a comprehensive understanding of the use of Kirkpatrick’s framework in nursing and the identification of gaps for future studies(19).

This integrative review was registered on the Open Science Framework (OSF) platform on May 20^th, 2022, and the protocol can be accessed at osf.io/uprv7, with the following DOI 10.17605/OSF.IO/JQ8U9(20).

Locus

This integrative review was conducted in the municipality of Thunder Bay, located in the province of Ontario (ON), Canada.

Period

The study period was September 2022 to November 2023.

Population

We used the review question: “Which is the evidence in the use of the Donald Kirkpatrick framework to evaluate nursing workplace training?” was used. The formulation of the research question was developed from the PCC strategy namely: Population (P): nursing; Concept (C): Donald Kirkpatrick’s four-level framework; Context (C): evaluation of workplace training.

Selection criteria

Studies were included if they addressed the research question and if they were published in English, Spanish, or Portuguese between 2006 to 2023, with full content available. The time frame was based on the year of publication of the third edition of the book “Evaluating training programs” where some interpretations of the traditional framework were updated(7). Studies that did not meet the inclusion criteria or did not clearly describe the instruments used were excluded. We also removed duplicates, literature reviews, and gray literature.

Sample definition

The search was performed in November 2022 from Thunder Bay, ON, Canada by FMM, end updated in August 2023. The Latin American and Caribbean Health Sciences Literature (LILACS), Medline (via PubMed), and Web of Science (WoS) bibliographic research databases were used.

Search strategies (strings) were based on the Health Sciences Descriptors (DeCS) and the Medical Subject Heading (MeSH). After analysis of exploratory searches, we decided to include the keyword “Kirkpatrick’’ to reduce the scope of the search. The final strategy used was: (”Education, continuing” OR “Continuous Learning” OR “Learning, Continuous” OR “Lifelong Learning” OR “Learning, Lifelong” OR “Life-Long Learning” OR “Learning, Life-Long” OR “Learnings, Life-Long” OR ’Life Long Learning” OR “Life-Long Learnings” OR “Continuing Education”) AND ((“nurses” OR ’Nurse” OR “Personnel, Nursing” OR “Nursing Personnel” OR “Registered Nurses” OR ’Nurse, Registered” OR “Nurses, Registered” OR “Registered Nurse”) OR (“nursing, team”)) AND (“Kirkpatrick”) for MEDLINE and WoS. For LILACS, it was decided to use only the keyword “Kirkpatrick” considering the theme specificity and the quantity of studies retrieved in the original string.

After searching the databases, the files generated in the BIBTXT or RIS format were exported to the Rayyan^® review manager(21) which supported the initial screening of abstracts and titles using a semi-automation process. The selection of articles was carried out in two blind stages by two different researchers (FMM and BVS); the first screening was completed by reading the titles and abstracts of the retrieved documents; the second screening involved a complete review. Conflicts were discussed until a consensus was reached. A third researcher (VAM) was available to decide any persistent conflict.

Data collection

The data was extracted using an analytical matrix built in Microsoft Excel^® containing information of the characteristics of the studies (authors, year of publication, country of publication, objective, type of study, sample and results) and information on the use of the Kirkpatrick model (levels used, elements, timing, evaluation criteria and measures).

Data treatment and analysis

We analyzed the data through reduction and comparison to identify patterns, themes, or relationships(19). Subsequently, a descriptive analysis based on the theoretical-conceptual framework adopted was carried out, considering the precepts of Donald Kirkpatrick’s framework(7).

This review followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for reporting systematic reviews(22). PRISMA was designed for systematic reviews of studies evaluating the effects of health interventions; however, the items on the checklist are applicable to reports of systematic reviews in other areas or evaluating other types of interventions(22). Thus, all PRISMA section were used to guide the review report, except for topics not applicable to this review’s approach, including analyses of the risk of bias, determination of measures of effect, and analyses of possible causes of heterogeneity. The effectiveness of the Kirkpatrick framework through the training effectiveness results shown in each article included in the sample selected for this study was analyzed. To do this, both the training design and the use of Kirpatrick levels were considered, the configuration of the instrument used, the timing of the evaluation and other aspects.

Results

One hundred and eight articles were retrieved and 13 were included in the final sample (Figure 1).

Figure 2 shows the 13 articles characterized by country, year of publication, objective and methods.

Figure 1 -
PRISMA(22) workflow for sample selection. Thunder Bay, ON, Canada, 2023

Figure 2 -
Sample characterization (n = 13). Thunder Bay, ON, Canada, 2023

The Kirkpatrick model has been useful in training, continuing education and updating courses, as well as in specialization programs with individual-centered and team-based interventions. Figure 3 shows the use of the four-levels Kirkpatrick model to evaluate training results:

Figure 3 -
Using the four-levels Kirkpatrick model to evaluate training results (n = 13). Thunder Bay, ON, Canada, 2023

Kirkpatrick’s framework was effective in evaluating all the training identified, considering the measurable and prolific outcomes of the samples by level in different types of training. Consistency in compliance with the guidelines was observed, demonstrating their versatility and flexibility.

Brazil was the only country with more than one publication(25-26,28-29). Quasi-experimental approaches(10,27-28,30-32) followed by qualitative research(24,26,33), mixed-methods research(23,25,29) and one experimental study(34) were found.

Most of the studies used all four levels of Kirkpatrick’s Framework(10,25,28,30,32-33) or a combination of the first three levels(23,26,29,31,34). One article evaluated the intervention with a combination of levels I and II(27) and another with a combination of levels III and IV(24).

Figure 4 shows a timeline for the implementation of these levels.

Figure 4 -
Multiple data points were collected before and after the intervention using the four levels of Kirkpatrick’s model (n = 13). Thunder Bay, ON, Canada, 2023

Two studies(28,33) did not identify the timeline for data collection. They only described that there was cross-sectional data collection considering the availability of the intervention for a period of time such as “since 2014”(28) or “between 2008 and 2011”(33). None of the studies had explicit follow-up designs, although some of them collected data long after the intervention(10,24-25,32). In one of them, the authors cited a previous data collection after the intervention, but did not present or compare the data, nor did they characterize a follow-up study(24).

Different instruments have been used to evaluate interventions, based on the specificity of the investigation; however, a main approach or a standard instrument for evaluating any level of the Kirkpatrick Framework was not found. Considering the relevance of measurement tools for the four levels of Kirkpatrick’s model and the lack of consensus on the best way to do it (and this is the richness of the model – it can be customized according to with study), it was decided to synthesize the instruments that were used in each selected study to help future nurses in this topic.

Quantitative instruments were the most common. Table 1 shows the quantitative instruments used to evaluate the training programs, their measures (items and range), and whether the study described some source of reliability using Cronbach’s α.

Table 1 -
Quantitative instruments applied to evaluate the four-level Kirkpatrick’s framework (n = 13). Thunder Bay, ON, Canada, 2023

Level: Instrument Description	Item	Range	α*
I: Five-point Likert scale for learner’s satisfaction(23)	5	1-5	n/s†
I: Five-point Likert scale’s questionnaire: “material and syllabus”, “assignments”, “tutoring”, and “learning support structure”(25)	4	1-5	n/s†
I: Reaction assessing tool including exogen aspects, (e.g., internet access) and endogenous aspects (e.g., material) for each module and the entire course(29)	9	1-4	n/s†
I: NOIC‡ with 5-point Likert scale(10)	21	1-5	0.94
I: ABC-SAT§, domains: a = affective, b = behavioral, c = cognitive(27)	a = 3	0-12	0.60
	b = 2	0-8	0.66
	c = 6	0-24	0.83
I: Five-point Likert scale statements related to satisfaction, potentiality for optimal learning and bad feelings’ prevention(28)	3	1-5	n/s†
I: Five-point Likert’s scale questionnaire, domains: c = content, t = teachers, f = facilities(30)	c = 5	1-5	0.87
	t = 4
	f = 3
I: Five-point Likert scale’s survey related to usability, format, discussion opportunities, participation and time adequacy(31)	6	1-5	n/s†
I: IMMS||(32)	4	1-5	0.79
I: Self-reported satisfaction questionnaire, domains c = “content and its relevance” and s = “support and facilitation”(34)	c = 35	0-35	0.92
	s = 15	0-15	0.77
II: 10-questions education knowledge and confidence test(23)	10	5-50	0.92
II: Four-Processual tests (one for each learning module)(25)	n/s†	0-10	n/s†
II: The innovation self-efficacy questionnaire(10)	8	1-5	0.86
II: Self-developed written knowledge test regarding key topics of the intervention, each with 4 possible answers(27)	9	0-18	n/s†
II: Five-point Likert scale’s statements: “(...) was used to increase my theoretical knowledge” and “(...) was used to increase my ability to solve clinical problems”(28)	2	1-5	n/s†
II: Awareness questionnaire(30)	15	0-15	0.87
II: 5-question closed-end test (2 had more than one correct answer)(31)	5	0-5	n/s†
II: Metacognition Questionnaire(32)	30	n/s†	n/s†
II: 1-hour knowledge test(34)	30	0-30	n/s†
II: Assignment submission requiring a description of an exercise program tailored to a hypothetical client scenario(34)	1	0-100	n/s†
III: Self-reported questionnaire about behavior regarding patient safety protocols(25)	9	1-5	n/s†
III: NIBS¶(10)	10	1-5	0.90
III: Two closed-ended statements about decreased of anxiety/stress and ability of solve a clinical problem/situation(28)	2	1-5	n/s†
III: The performance questionnaire(30)	15	0-15	0.78
III: Self-reported Knowledge transfer at workplace questionnaire(32)	14	1-5	0.94
III: Self-reported satisfaction questionnaire, domain b = “change in clinical behavior”(34)	b = 30	0-30	0.84
IV: Questionnaire of agreement about the implantation of five goals for patient safety workplace after the course(25)	6	1-5	n/s†
IV: Scale of Clinical Nursing Staff Innovation Ability(10)	41	0-205	0.82
IV: The questionnaire for exposure to sharp objects, blood and body fluids(30)	53	n/s†	0.89

• *α = Cronbach’s α; ^†n/s = Not specified; ^‡NOIC = The Nurse Organizational Innovation Climate Scale; ^§ABC-SAT = The ‘Affective-Behavioral-Cognitive-Satisfaction questionnaire’; ^||IMMS = The Instructional Materials Motivation Survey; ^¶NIBS = Nurse Innovation Behaviour Scale

A great variability of instruments regarding quantitative approaches was found. Level I was mostly evaluated by a five-point Likert scale for items such as satisfaction(23,25,27-28,30,32,34), immersion(28,31), climate(10), relevance and confidence(32). Both knowledge and awareness approaches were used on level II. The most common was closed-end learning tests(25,27,29-31,34), followed by self-rated questionnaires(10,23,32). Level III and IV quantitative measures included trainees’ perception of knowledge transfer to the workplace through closed-end questionnaires(10,25,28,30,32-34) or scales(10).

Although less frequent than quantitative, the use of a qualitative approach was found in behavior level(23-24,26,29,31,33) followed by reaction(26,29,33), results(24,28,33) and learning(26,33). Seven(23-24,26,28-29,31,33) studies used qualitative instruments to evaluate training programs, however only two(24,26) of them did so without a quantitative piece. Semi-structured interviews(23-24,26), questionnaires with open questions(28,33), descriptive analysis of the key words in students’ projects(29), qualitative research(31), qualitative analysis based on the participants’ narratives(29), and adherence after the first month of the intervention(29) were identified. The most prevalent type of qualitative analysis was thematic analysis(24,26,28-29). Most of the studies did not describe reliability or validity of the qualitative instruments, with the exception of Koto-Shimada(24) who carried out a pilot test. Other work-related measures have been also reported, i.e., clinical records and patient information(32), self-reported exposure to needlestick injury questionnaire(30).

Choosing a measurement tool is crucial to evaluating the effectiveness of any intervention, especially when it comes to training. This step is embodied into Kirkpatrick’s model and should be carefully selected by nurses in order to qualify the analysis of each level. The more precise the tool (i.e., more in line with the object of study), the greater the chances of a reliable evaluation.

Discussion

This review has shown that the Kirkpatrick Framework has been widely used in the evaluation of nurses’ training and also in other areas of health(11). Our sample depicted in this study indicated consistency in compliance with the guidelines and the main training outcomes indicate that this model is useful to evaluate effectiveness, with positive results at each level in different types of training interventions.

The evaluation process converged with the orientation of the framework which focuses on participant outcomes(7). However, one study approached the participants’ managers and a complementary view emerged, as exemplified in the quote: “the nurses did not experience the change in the atmosphere and attitudes in the same way as the nurse managers had observed”(33). The inclusion of the leadership perspective can be positive, especially to obtain feedback on knowledge transfer to the workplace (level III) or changes in practice or organizational outcomes (level IV) considering their supervisory role.

The most common type of baseline information was on Learning level(10,23,27,30,32). The Behavior and Results levels were the most divergent levels regarding timing. There are no guidelines in Kirkpatrick’s Framework on the best time to evaluate each level(7,15) and we cannot support a more common or assertive time for data collection, considering that none of the studies converge on the best time. However, it is important to point out that training programs expect short-, medium- and long-term outcomes. Therefore, selecting a specific time (e.g., before, during, soon after, longer after) or period (e.g., follow-ups) is essential for a comprehensive evaluation that considers that feeling of reaction will fade over time and the most important outcomes will appear over time(8).

The Kirkpatrick model is a flexible and adjustable framework that embeds different tools for evaluating training outcomes (Reaction, Learning, Behavior and Results), which reinforces its power, since the realities, training and contexts of each organization are completely different. However, precisely because it does not indicate a specific methodological standard, authors must be careful to ensure the validity and reliability of the information collected(17).

Both the number of studies and the diversity of instruments decreased as the level of evaluation increased, it has been observed. The literature review indicates that the Results’ level is the most difficult to achieve. There is acknowledged that level IV “could identify the added value to society of a given educational program, as it makes it possible to evaluate not only of the application of intervention projects in the practice of health professionals, but also the results of their use in a local context”(29). However, half of the samples in this review did not report measuring this level. The most common reason reported for the exclusion of level IV was the need for medium- and long-term follow-up periods(26) and the complexity of relating the outcomes with the training program, distinguishing them from many other factors that could affect the results(31). Results evaluation (level IV) is the most challenging because it must consider the transformation or impacts of training on organizational practice, which requires other types of instruments, organizational information and the point of view of other stakeholders, as well as more time. In this sense, similar data was found regarding the perceived relevance of levels III and IV and the difficulty to implement these levels(17).

There is some criticism about the rigidity of this framework, which can lead to essential aspects of the evaluation, such as the formative vision being missed(16,18). Only one study reported data collection in each module and at the end of the course at the reactions level(29). No other experience was identified at the follow-up. This evidence shows that, despite the diversity in the perspective of outcomes evaluation tools, especially with the use of qualitative approaches to capture the trainees’ perspective, all the experiences reported were in a summative or transversal view.

Trainers are not using only the highest levels, which is a concern of the criticism of Kirkpatrick’s model(16-18) it has been observed. Six articles managed to reach Level four and another five used a combination of the first three levels; accounting for 85% of the sample. This indicates that nursing trainers are attempting to use all four levels of the model to collect information on outcomes, despite the challenge regarding the Results level. This reaffirms the researchers’ commitment to the framework’s recommendation to use a complete model for greater accuracy(7-8).

The relevance of this review is the fact that it synthesizes the recognition of the operationalization of the framework. A large bibliometric analysis found the Kirkpatrick model to be a trending topic of interest after the 2000s, including 20.7% of its sample as some kind of literature review publications. However, most studies aimed to criticize the model or identify its benefits, and little review evidence was found on the types and characterization of evaluation tools and their application in a specific field, such as nursing(11).

This study has methodological limitations. The multiple research designs of the publications included made it impossible to produce evidence through meta-analysis or meta-synthesis techniques. Considering the nature of the integrative review, the aim is not to critically appraise the quality of the studies and therefore, the quality of the study was not used an inclusion criterion.

We encourage further research to identify the best constructs for evaluating each level, to expand the implementation of level IV and to foster scientific conversation about the methodological orientation for composing the chain of evidence between the levels, as proposed by the New World Kirkpatrick model(15). Moreover, our literature review identified limited longitudinal studies with consecutive follow-ups for data collection, which is highly recommended for achieving high-level evidence regarding causal pathways.

Conclusion

Kirkpatrick’s four-level Framework is a suitable choice in terms of adapting evaluation instruments to the training design. The selection of a reliable set of instruments is crucial to qualify the analysis of each level, with Likert scales being the most common choice for trainers. This model is prolific for evaluating training in the nursing field but requires a careful choice of instruments for each level. Many qualitative and quantitative measurement tools have been identified that can be useful for practitioners and academics in further evaluation and research.

References

• Nahm ES, Archibald M, Mills ME, Costa L, Warren J, Nair P, et al. Continuum of nursing education and practice: Time to close the chasm between academia and practice. J Prof Nurs. 2023;46:134-40. https://doi.org/10.1016/j.profnurs.2023.02.012
  » https://doi.org/10.1016/j.profnurs.2023.02.012
• Buljac-Samardzic M, Doekhie KD, van Wijngaarden JDH. Interventions to improve team effectiveness within health care: a systematic review of the past decade. Hum Resour Health. 2020;18(2). https://doi.org/10.1186/s12960-019-0411-3
  » https://doi.org/10.1186/s12960-019-0411-3
• Ertan ŞS, Şeşen H. Positive organizational scholarship in healthcare: The impact of employee training on performance, turnover, and stress. JMO. 2022;28(6):1301-20. https://doi.org/10.1017/jmo.2019.61
  » https://doi.org/10.1017/jmo.2019.61
• Pantoja T, Opiyo N, Ciapponi A, Dudley L, Gagnon MP, Herrera CA, et al. Implementation strategies for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9(9):CD011086. https://doi.org/10.1002/14651858.CD011086.pub2
  » https://doi.org/10.1002/14651858.CD011086.pub2
• Rivaldo Y, Nabella SD. Employee Performance: Education, Training, Experience and Work Discipline. Qual Access Success. 2023;24(193):182-8. https://doi.org/10.47750/QAS/24.193.20
  » https://doi.org/10.47750/QAS/24.193.20
• Ogata MN, Silva JAM, Peduzzi M, Costa MV, Fortuna CM, Feliciano AB. Interfaces between permanent education and interprofessional education in health. Rev Esc Enferm USP. 2021;55:e03733. https://doi.org/10.1590/S1980-220X2020018903733
  » https://doi.org/10.1590/S1980-220X2020018903733
• Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publisher; 2006.
• Yardley S, Dornan T. Kirkpatrick’s levels and education ‘evidence’. Med Educ. 2012;46(1):97-106. https://doi.org/10.1111/j.1365-2923.2011.04076.x
  » https://doi.org/10.1111/j.1365-2923.2011.04076.x
• Heydari MR, Taghva F, Amini M, Delavari S. Using Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff. BMC Res Notes. 2019;12(1):388. https://doi.org/10.1186/s13104-019-4421-y
  » https://doi.org/10.1186/s13104-019-4421-y
• Huang X, Wang R, Chen J, Gao C, Wang B, Dong Y, et al. Kirkpatrick’s evaluation of the effect of a nursing innovation team training for clinical nurses. J Nurs Manag. 2022;30(7):2165-75. https://doi.org/10.1111/jonm.13504
  » https://doi.org/10.1111/jonm.13504
• Alsalamah A, Callinan C. The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020). Ind Commer Train. 2022;54(1):36-63. https://doi.org/10.1108/ICT-12-2020-0115
  » https://doi.org/10.1108/ICT-12-2020-0115
• Choy CL, Liaw SY, Goh EL, See KC, Chua WL. Impact of sepsis education for healthcare professionals and students on learning and patient outcomes: a systematic review. J Hosp Infect. 2022;122:84-95. https://doi.org/10.1016/j.jhin.2022.01.004
  » https://doi.org/10.1016/j.jhin.2022.01.004
• Khurshid Z, De Brún A, McAuliffe E. Protocol for an integrated evaluation framework to study training, curricular and contextual factors impacting the success of a measurement for improvement training programme for healthcare staff in Ireland. BMJ Open. 2022;12(2):e047639. https://doi.org/10.1136/bmjopen-2020-047639
  » https://doi.org/10.1136/bmjopen-2020-047639
• Malhotra S, Negi G, Sharma SK, Kaur R. A prospective interventional study to assess the impact of a ‘structured compact training’ on knowledge and skills of safe blood transfusion practices among nurses working in a tertiary care institute. Transfus Med. 2022;32(1):32-7. https://doi.org/10.1111/TME.12835
  » https://doi.org/10.1111/TME.12835
• Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s Four Levels of Training Evaluation. 1st ed. Alexandria, VA: Association for Talent Development; 2016.
• Reio TG Jr, Rocco TS, Smith DH, Chang E. A Critique of Kirkpatrick’s Evaluation Model. New Horiz Adult Educ. 2017;29(2):35-53. https://doi.org/10.1002/nha3.20178
  » https://doi.org/10.1002/nha3.20178
• Rosa A. Kirkpatrick’s four-level model for evaluating training programs: the weak links in a “chain of evidence”. LLL. 2018;14(31):17-37. https://doi.org/10.19241/lll.v14i31.113
  » https://doi.org/10.19241/lll.v14i31.113
• Cahapay M. Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation. Int J Assess Tool Educ. 2021;8(1):135-44. https://doi.org/10.21449/ijate.856143
  » https://doi.org/10.21449/ijate.856143
• Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546-53. https://doi.org/10.1111/j.1365-2648.2005.03621.x
  » https://doi.org/10.1111/j.1365-2648.2005.03621.x
• Miranda FM, Mininel VA. Use of Kirkpatrick’s framework to evaluate nursing team’s training: integrative literature review protocol. Open Science Framework. 2023. https://doi.org/10.17605/OSF.IO/JQ8U9
  » https://doi.org/10.17605/OSF.IO/JQ8U9
• Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:210. https://doi.org/10.1186/s13643-016-0384-4
  » https://doi.org/10.1186/s13643-016-0384-4
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71
  » https://doi.org/10.1136/bmj.n71
• Parmar JK, L’Heureux T, Anderson S, Duggleby W, Pollard C, Poole L, et al. Optimizing the integration of family caregivers in the delivery of person-centered care: evaluation of an educational program for the healthcare workforce. BMC Health Serv Res. 2022;22(1):364. https://doi.org/10.1186/s12913-022-07689-w
  » https://doi.org/10.1186/s12913-022-07689-w
• Koto-Shimada K, Fujita N, Matsuoka S, Jimba M, Touch S, Zwi AB. Medium-term outcomes of a program to upgrade the nursing faculty in Cambodia: A qualitative study. Nurse Educ Today. 2022;116:105438. https://doi.org/10.1016/j.nedt.2022.105438
  » https://doi.org/10.1016/j.nedt.2022.105438
• Santos RA, Mendes WV Júnior, Martins M. Quality in healthcare and patient safety: evaluation of the results of a distance learning program. Cien Saude Colet. 2021;26(10):4553-68. https://doi.org/10.1590/1413-812320212610.11082021
  » https://doi.org/10.1590/1413-812320212610.11082021
• Machado MGO, Sampaio CL. Training on common mental disorders at a hospital ward: the use of active methodologies in care construction. SMAD, Rev Eletrônica Saúde Mental Álcool Drog. 2021;17(1):75-83. https://doi.org/10.11606/issn.1806-6976.smad.2021.168134
  » https://doi.org/10.11606/issn.1806-6976.smad.2021.168134
• Jobst S, Windeisen M, Wuensch A, Meng M, Kugler C. Supporting migrants and refugees with posttraumatic stress disorder: development, pilot implementation, and pilot evaluation of a continuing interprofessional education for healthcare providers. BMC Med Educ. 2020;20(1):311. https://doi.org/10.1186/s12909-020-02220-3
  » https://doi.org/10.1186/s12909-020-02220-3
• Knobel R, Menezes MO, Santos DS, Takemoto MLS. Planning, construction and use of handmade simulators to enhance the teaching and learning in Obstetrics. Rev. Latino-Am. Enfermagem. 2020;28:e3302. https://doi.org/10.1590/1518-8345.3684.3302
  » https://doi.org/10.1590/1518-8345.3684.3302
• Castro JA Filho, Motta LB. Evaluation in distance learning: a case report of the UNASUS/UERJ postgraduate course in elderly health. Rev Bras Geriatr Gerontol. 2018;21(5):513-22. https://doi.org/10.1590/1981-22562018021.180021
  » https://doi.org/10.1590/1981-22562018021.180021
• Bijani M, Rostami K, Momennasab M, Yektatalab S. Evaluating the Effectiveness of a Continuing Education Program for Prevention of Occupational Exposure to Needle Stick Injuries in Nursing Staff Based on Kirkpatrick’s Model. J Natl Med Assoc. 2018;110(5):459-63. https://doi.org/10.1016/j.jnma.2017.11.002
  » https://doi.org/10.1016/j.jnma.2017.11.002
• LaMar RA. Using a Virtual Journal Club for Sharing Evidence-Based Practice Recommendations in Critical Care Registered Nurses. Teach Learn Nurs. 2017;12(1):53-8. https://doi.org/10.1016/j.teln.2016.08.006
  » https://doi.org/10.1016/j.teln.2016.08.006
• Liaw SY, Wong LF, Lim EYP, Ang SBL, Mujumdar S, Ho JTY, et al. Effectiveness of a Web-Based Simulation in Improving Nurses’ Workplace Practice With Deteriorating Ward Patients: A Pre- and Postintervention Study. J Med Internet Res. 2016;18(2):e37. https://doi.org/10.2196/jmir.5294
  » https://doi.org/10.2196/jmir.5294
• Lahti ME, Kontio RM, Välimäki M. Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views. Perspect Psychiatr Care. 2016;52(1):40-8. https://doi.org/10.1111/ppc.12100
  » https://doi.org/10.1111/ppc.12100
• Maloney S, Haas R, Keating JL, Molloy E, Jolly B, Sims J, et al. Effectiveness of Web-Based Versus Face-To-Face Delivery of Education in Prescription of Falls-Prevention Exercise to Health Professionals: Randomized Trial. J Med Internet Res. 2011;13(4):e116. https://doi.org/10.2196/jmir.1680
  » https://doi.org/10.2196/jmir.1680

• How to cite this article
  Miranda FM, Santos BV, Kristman VL, Mininel VA. Employing Kirkpatrick’s framework to evaluate nurse training: an integrative review. Rev. Latino-Am. Enfermagem. 2025;33:e4431 [cited yearmonth day]. Available from: URL . https://doi.org/10.1590/1518-8345.7250.4431
• *
  Paper extracted from doctoral dissertation “Training Brazilian nurses to promote mental health at work”, presented to Universidade Federal de São Carlos, São Carlos, SP, Brazil. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) - Finance Code 001, Brazil.

Edited by

• Associate Editor:
  Maria Lúcia Zanetti

Publication Dates

• Publication in this collection
  03 Feb 2025
• Date of issue
  2025

History

• Received
  20 Dec 2023
• Accepted
  07 Aug 2024