The effect of training in Integrated Management of Childhood Illness (IMCI) on the performance and healthcare quality of pediatric healthcare workers: a systematic review

Objectives: to analyze the effect of training in Integrated Management of Childhood Illness (IMCI) on the quality of case management by healthcare workers based on a systematic review of the literature. Methods: the authors searched the databases MEDLINE, LILACS, PAHO and WHOLIS for the search terms Integrated Management of Childhood Illness (IMCI), and analyzed documents published by Pan American Health Organization, World Health Organization and the Brazilian Ministry of Health between January 1993 and July 2006. The quality of the methodology was assessed using the criteria developed by Downs and Black. Results: thirty-five papers were reviewed. Twelve of these validated the IMCI algorithm and found the sensitivity to be high and the specificity to be over 80% for major illnesses. Twenty-three papers assessed the performance of healthcare workers, eight of these with no control group. The present study shows clear evidence of improvement in the performance of healthcare workers employed at healthcare facilities with IMCI. The main methodological weaknesses of the study were lack of control of confounding factors and lack of information regarding statistical power. Conclusions: the performance of healthcare workers tends to improve at public healthcare facilities when IMCI is introduced.

provided references for additional studies on IMCI, and unpublished articles of acceptable quality were included.The search covered the period from January 1993 (when IMCI was launched) to July 2006.
Studies were included if they contained quantitative data comparing the performance of healthcare workers trained in IMCI to that of workers with no such training.Uncontrolled studies on the performance of health workers trained in IMCI were also eligible.Studies measuring the impact of IMCI on health indicators but not the performance of health workers were not included.Also excluded were studies using an exclusively qualitative methodology, manuals, technical information packs and reports providing no evaluation of the strategy.
Analytical experimental or observational designs were included.The outcome parameters included performance indicators of health workers in evaluation, classification, treatment and counseling of mothers.
Subsequently a critical evaluation of the selected articles was carried out, using the 27 quality criteria proposed by Downs and Black, including communication, external validity, internal validity (bias), internal validity (confounding factors) and statistical power. 6These criteria relate to positive, partially positive or negative responses to questions on the methodological features of the article, such as whether the statistical tests used to measure the main results were adequate.
The responses to the 27 quality criteria were entered into an Excel spreadsheet.A methodology score was used to indicate the quality of each article with regard to the strength of the evidence by dividing the number of positive items by the number of items evaluated.
Data collection was carried out using an article extraction form requesting the following informa-

Introduction
The Integrated Management of Childhood Illness (IMCI) strategy was drawn up by the World Health Organization (WHO) in collaboration with the United Nations Children's Fund (UNICEF) with the aim of improving child health indicators.The global strategy was launched in 1993 1,2 and was introduced in Brazil in 1996, initially in the North and Northeast states where conditions for child health are the least favourable. 3he IMCI strategy includes training of health workers in the management of diseases common in childhood, with emphasis on diarrhea, respiratory infections, malaria, measles and malnutrition.It also includes support for health services that deal with the prevention of specific diseases and health promotion.The IMCI training course originally lasted 11 days, but in a number of countries and locations the course has been reduced to seven or eight days. 4lthough the IMCI strategy has already been introduced in more than 100 countries, 5 so far no systematic review of the literature has evaluated whether there has been an improvement in the performance of health workers subsequent to the introduction of this strategy.Such information would be extremely useful for planners and policy-makers in the area of child health.The present review summarises the literature on the quality of case management by health workers trained in IMCI.

Methods
The systematic review of the literature on the IMCI strategy included studies directly or indirectly dealing with the question of whether the training of health workers in IMCI has resulted in an adequate performance in the management of the main health problems arising in childhood.
The publications were located and selected using electronic medical science databases -MEDLINE (National Library of Medicine) and LILACS (Latin American Literature on Medical Science) -in addition to the databases of the international organizations PAHO (Pan American Health Organization) and WHO (World Health Organization), using the following key words: AIDPI (Atenção Integrada às Doenças Prevalentes da Infância), IMCI (Integrated Management of Childhood Illness) and AIEPI (Atención Integrada a las Enfermedades Prevalentes de la Infancia).The review was broadened to include other sources.Thus, officials from health institutions (PAHO, WHO and the Brazilian Ministry of Health) Integrated Management of Childhood Illness on the performance of healthcare workers bases contained the key words AIDPI, IMCI or AIEPI.In addition, seven documents published by the WHO and PAHO between January 1993 and July 2006 were included.
Of these documents (manuals, technical information packs, reports and articles) 33 met the inclusion criteria of the study.The studies had all been carried out by researchers working in developing countries (17 in Africa where the strategy was first tested and put into practice).
A list of these 33 articles can be found in Table 1 (preliminary studies evaluating the IMCI algorithm with a view to validation), Table 2 (studies evaluating the performance of healthcare workers trained in IMCI with no external control group) and Table 3 (studies with an external control group evaluating the performance of healthcare workers trained in IMCI).
The principal methodological problems encountered included failure to control for confounding factors in studies with control groups and failure to determine the statistical power of most of the uncontrolled studies.Given the variety of problems detected, the methodology score for the articles varied from 0.50 to 1.00.
Fourteen articles dealt with the validation of specific aspects of the IMCI algorithm, comparing the diagnostic studies carried out by health workers trained in the strategy with the gold standard set by experienced pediatricians.][9][10][11][12][13][14][15][16][17][18][19][20] The main exception was the low sensitivity of pallor in the palms as an indicator of anemia. 10,16Two authors reported low specificity for signs indicating the need for hospital referral, 11,18 while another 17 observed a specificity of 74%.
2][23][24][25][26][27][28][29] Some cross-sectional studies measured performance after training and reported absolute levels of adequacy, 21,[23][24][25][26] with, for example, 80% of healthcare workers asking about immunizations.Others compared the performance of the same health workers before and after training. 28,29Generally, these studies showed high levels of satisfactory performance, with a number of exceptions, including the correct treatment for anemia, 23,24 evaluation of general danger signs, 21,23 evaluation of the health of carers, 26,27 or the need to return immediately to the health unit. 21,23he study conducted by Kelley et al. 22 in Uganda, which also falls into this category, compared the performance of healthcare workers who received training in IMCI plus immediate feedback from other trained workers to the performance of trainees not receiving such feedback, showing that feedback improves performance.
Out of ten articles evaluating the performance of healthcare workers in comparison with an external control group, [30][31][32][33][34][35][36][37][38][39] five were of high quality with a methodology score above 0.9, [31][32][33][34][35] the others receiving scores between 0.5 and 0.86. 30,33,36,37he studies with an external control group included both studies of efficacy, in which healthcare workers were specially trained by the researchers and subsequently evaluated, and studies of effectiveness, in which previously trained workers were evaluated under routine medical care conditions.[34][35] The study in Uganda showed a significant impact on several of the items studied, but differences between trained and untrained healthcare workers were slight.38 This may be attributed to the short duration of the training process and/or the use of poorly qualified instructors in an attempt to achieve as wide a coverage as possible.  The orithm covers most diseases (92%).
For hospitalized cases, sensitivity for general warning signs was low (39%), while specificity was high (87%).With regard to criteria for referral, sensitivity and specificity were 81% and 74%, respectively.
The IMCI algorithm reduced missed opportunities for vaccination by 50%.Sensitivity greater than 60% and specificity greater than 85%.For the classification of dengue fever, the specificity was between 50 and 55% in children with serum testing positive for dengue.Staff showed a high degree of adherence to the algorithm for most of the items studied.

0.79
Performance was weak for advice on medication and asking questions about the mothers' health problems.Compared to the control group, there was a significant improvement in the performance of staff, mothers' knowledge and paractices, and weight gain in children.

Table 3
Studies of improvements in the performance of Integrated Management of Childhood Ilness (IMCI) trained healthcare workers, with an external control group.monia based on breathing frequency, without the use of radiology or auscultation.
The second category of studies was carried out to evaluate the performance of health workers with or without a control.The uncontrolled studies evaluated the adequacy of the performance, 40 that is, whether IMCI-trained healthcare workers had high levels of performance for the evaluation, classification and counseling of children and their carers.These studies provided evidence of adequate performance levels among healthcare workers trained in IMCI, 7 showed that feedback from other trained staff improved performance further, 21,[23][24][25][26] and that IMCI is important for counseling carers. 27One study suggested that the IMCI improves care for sick children, but only if the health facility has a good infrastructure. 29hese findings were confirmed in studies involving an external control group, where the methodology is more sophisticated and allows to determine the likelihood of the observed effect being due to IMCI training. 40,41These studies provided evidence of improved performance among healthcare workers trained in IMCI, 30,32,34,[37][38][39] a positive effect of IMCI on nutritional status, 31 better communication in the group trained in IMCI 33 and adequate

Discussion
The studies reviewed fall basically into three categories: 1) studies for initial validation of the IMCI algorithm, 2) studies without an external control group, evaluating the performance of trained healthcare workers and 3) studies with a control group, comparing trained and untrained healthcare workers.
The first studies carried out were validation studies, as it was necessary to evaluate the discriminatory power of the IMCI algorithm before introducing it into other countries.These studies provided evidence of the ability of IMCI to detect nutritional problems 7 and to identify unvaccinated children 17 and seriously ill children in need of referral, 11,15,18 and showed IMCI to be useful in the management of the main childhood health problems 9,13,14,19 with adequate levels of sensitivity and specificity. 8,12The main exception was the diagnosis of anemia based on palmar pallor which was associated with low to moderate accuracy. 10,16These preliminary studies were very useful in that they laid the foundations for the strategy and gained respectability for the algorithm in the medical community, which was initially somewhat skeptical of some of the procedures included in the IMCI, such as the diagnosis of pneu-conclusion the most common.Although their statistical power was not specified, the studies that evaluated performance used samples of more than 100 children, and, therefore, are sufficiently powerful statistically to detect important differences.][33][34][35][36] In conclusion, the studies evaluated show scientific evidence of improvement in child healthcare in facilities with staff trained in IMCI in terms of advice on nutrition, weight gain, knowledge of children's health, correct treatment, evaluation of general danger signs, and correct prescription of antibiotics.More studies are required to clarify whether this is also the case for treatment of diarrhea plus dehydration, detection and treatment of anemia and earache.Localities where IMCI has not yet been introduced, or has been only partially introduced, should be encouraged to adopt the strategy.
In view of the conclusions of this systematic review, some recent developments in Brazil give cause for reflection.The first is the reduction in the pace of implementation of IMCI in the country owing to a change in the priorities of the Children's Health Department of the Ministry of Health.The second relates to the directive that medicines be prescribed only by a qualified physician, with nurses being responsible for health promotion and detection of signs of disease (or of the risk of disease).Finally, much effort is currently being invested in the establishment of adequate healthcare for children integrating of primary care health workers in a more complex system of reference.
Healthcare policies should be strongly based on scientific knowledge.In that respect, the findings of the present systematic review of the literature show that training in IMCI has led to significant improvements in children's health in Brazil.use of antimicrobial agents. 36here was considerable variation between the studies with regard to study design, outcome parameters, target population and sample size.It was therefore not possible to obtain an overall average using meta-analytical techniques.However, this did not significantly alter the fact that most of the studies showed a positive association between IMCI training and performance, including those studies with a high methodology score.Two of the studies described here 36,37 included overall analyses of investigations carried out for the Multi-Country of IMCI Evaluation, which are tantamount to a meta-analysis.
It is important to point out that many different types of healthcare workers were trained in IMCI.In African countries, most were intermediate-level medical or nursing assistants belonging to various categories with more than 18-36 months training.A recent evaluation showed that training in IMCI improved the performance of healthcare workers at all levels, including qualified physicians. 42In Brazil, nurses trained in IMCI had a performance equal to or better than that of doctors trained in IMCI. 34ll but six studies were English language publications. 21,23,24,26,30,31It is possible that non-native English-speaking authors preferred to publish in that language, as clearly shown in one case. 34It should also be borne in mind that some publishers of English-language periodicals discriminate against work submitted by researchers from under-developed countries. 43ne problem to be considered is the bias of the publication, given that there is a greater likelihood of articles being accepted if they present positive rather than negative results. 44It should be noted that this review identified three unpublished studies, all presenting positive results. 21,25,30s for the methodological problems encountered, according to the criteria developed by Downs and Black, failure to describe the statistical power and failure to control for confounding factors were and specificity (fever).The combination of fever and mother's perception of short breath resulted in a more sensitive criterion than that of the present model for detecting cases of bacterial infection.Antibiotics were prescribed to 78% children with bacterial infection.
trained staff improved only during the early stages.Classification and management were unsatisfactory for pneumonia and for sick newborns.The performance was better in the case of fever, measles, dysentery and diarrhea.was a significant improvement in the evaluation of general warning signs, rational prescription of medication and early treatment at the clinic.There was no change in terms of advice on mediation or on when to return to the health facility.The health facilities had good infrastructure.0.83 in the IMCI group.Information on drug dosage was more adequate, carers understood the counseling given and the prescribed dosage.was associated with improvements in the evaluation and classification of children, and in counseling mothers; The algorithm had a moderate effect on indicators relating to adequate treatment.Supervision was inadequate in most states.differences between the groups at baseline; 18 months after training the average evaluation index was 73 for the IMCI group and 17 for the control group (out of a maximum of 100)by staff trained in IMCI had a greater likelihood of receiving adequate prescriptions of antibiotics than children treated by the control group.

38 Main author, country and year of publication Design and sample size Parameters Results Score
Sensitivity and specificity above 80% for common diseases.Some diseases, such as skin rash, mouth and eye disorders, were not covered by IMCI.

Table 1
Studies involving preliminary evaluation of the Integrated Management of Childhood Illness (IMCI) algorithm.Comparison of diagnosesgiven by IMCI-trained health workers and a gold standard.
continues Integrated Management

Table 1
Studies involving preliminary evaluation of the Integrated Management of Childhood Illness (IMCI) algorithm.Comparison of diagnosesgiven by IMCI-trained health workers and a gold standard.

Table 2
Studies of the performance of Integrated Management Childhood Illness (IMCI) trained healthcare workers without an external control group.
continues Integrated Management

Table 2
Studies of the performance of Integrated Management Childhood Illness (IMCI) trained healthcare workers without an external control group.

Table 3
Studies of improvements in the performance of Integrated Management of Childhood Ilness (IMCI) trained healthcare workers, with an

Table 3
Studies of improvements in the performance of Integrated Management of Childhood Ilness (IMCI) trained healthcare workers, with an external control group.