The impact of health-management training programs in Latin America on job performance

A study was undertaken in Mexico, Colombia, and El Salvador to determine the impact of a management training program on health managers’ job performance. A quasi-experimental design was used where in the baseline study an intervention group of 85 district health managers in the three countries was compared with a control group of 71 managers who did not receive the training program. After the implementation of an 18-month training program (which included 5-day training workshops and a series of tasks to be carried out between the workshops), the outcome in terms of improved job performance (i.e. use of predefined management techniques) was measured through twelve management performance indicators. The data collection tools were two questionnaires, participant observation in managers’ workplaces, focus group discussions, staff interviews, and document analysis. In Mexico, the control group showed 8.3 times weaker management performance compared to the intervention group; in Colombia the value was 3.6 and in El Salvador 2.4. Factors associated with a successful training outcome were: (a) training techniques, (b) strengthening of enabling factors, and (c) reinforcement mechanisms. Health Services Administration; Management Capacity; Management; Training Introduction Ongoing health reforms, which include decentralizing the sector in Latin America and elsewhere, have increased the importance of district health management. A number of authors 1,2,3,4,5 have stressed the importance of good management at the district level and the need for training. There is a broad literature on how to measure the impact of health education and management training 6,7,8. The information about the training outcome of courses related to the management of specific diseases is available, but in most cases the outcome indicators were “correct use of standard treatments” 9,10,11. In only very few cases, control groups were used and the measurement of change was generally done shortly after the end of the training 12,13. Much information is given about how the training was done and about participant satisfaction 14,15,16,17,18, but no studies on the impact of management training on job performance were identified. No rigid evaluation criteria or performance indicators were applied, and only one study 19 used a control group to compare management skills acquired in the training course to similar staff members who did not take part in the training. It was the objective of this research to measure the direct results of a new 18-month training program on district health managers’ knowlTHE IMPACT OF HEALTH-MANAGEMENT TRAINING PROGRAMS ON JOB PERFORMANCE 1111 Cad. Saúde Pública, Rio de Janeiro, 20(4):1110-1120, jul-ago, 2004 edge and performance in three Latin American countries.


Introduction
Ongoing health reforms, which include decentralizing the sector in Latin America and elsewhere, have increased the importance of district health management.A number of authors 1,2,3,4,5 have stressed the importance of good management at the district level and the need for training.
There is a broad literature on how to measure the impact of health education and management training 6,7,8 .The information about the training outcome of courses related to the management of specific diseases is available, but in most cases the outcome indicators were "correct use of standard treatments" 9,10,11 .In only very few cases, control groups were used and the measurement of change was generally done shortly after the end of the training 12,13 .Much information is given about how the training was done and about participant satisfaction 14,15,16,17,18 , but no studies on the impact of management training on job performance were identified.No rigid evaluation criteria or performance indicators were applied, and only one study 19 used a control group to compare management skills acquired in the training course to similar staff members who did not take part in the training.
It was the objective of this research to measure the direct results of a new 18-month training program on district health managers' knowl-edge and performance in three Latin American countries.

Methodology
Study areas and study design • Study groups: the study areas and groups of health managers attached to the Ministry of Health (MoH) and Social Security System (SSS) in three Latin American countries are shown in Table 1.
• Decision on the intervention-control design: ideally, a sample of managers from different regions should have been assigned randomly to the intervention group (with a training program) and to the control group (without the training program or with a different intervention).Unfortunately, random allocation in reallife situation where a health manager's training program is envisaged is not feasible.Therefore a quasi-experimental design was chosen.Health managers who experienced the training program were compared to a control group with similar characteristics but without the educational intervention.Both the baseline studies and the follow-up studies (after the training program for the intervention group) were undertaken among the intervention and control groups.In Mexico, the intervention and control groups were followed through the whole process, but in Colombia and El Salvador a second group of untrained managers had to be selected from the same management teams for the follow-up assessment.The first and second control group had the same general characteristics in terms of kind of posts, educational level, and time of employment (p < 0.05) and there were no statistically significant differences in indicators measuring job performance 20 between them.
Baseline study and outcome measurement in the follow-up study • Measuring health managers' knowledge: this was done in each of the training workshops by using knowledge tests with ten open-ended questions related to needs assessment, planning, communication, systems management, monitoring, and evaluation (see below).
• Measuring job performance: the quantitative baseline study before the start of the training program collected information about health managers' job performance; it included two self-applied questionnaires, which were filled in by all participants in the intervention and control groups.Internal consistency checks showed a consistency of 88.6% between key interview answers in the two questionnaires.The questionnaires were piloted in Chile and slightly modified.Twelve management performance indicators based on a literature review 20 measured if district health managers used essential management techniques or not; they included: (a) Needs assessment (identifying health needs of different population groups): 1. Use of health indicators and health services indicators; 2. Prioritization of problems for taking actions; 3. Identification of risk groups (risk assessment).(b) Planning: 4. Design of district health plans; 5. Implementation of these plans; 6. Use of visual aids (e.g.maps for specific programs or risk maps for identifying priority areas).The external consistency between quantitative and qualitative study showed a high reliability of data.
• Measuring outcome in the follow-up study: 18 months after the start of the training program, the same questionnaires were applied to the intervention and control groups in the three countries.The twelve indicators were also used for the outcome measure additionally to the information obtained in the qualitative study.
• Sample size calculation: The expected increase of adequate management performance (in terms of use of key management techniques) due to the educational intervention was from 60% beforehand to 95% afterwards, or from 60% in the control group to 95% in the intervention group after the training program.In order to find a significant difference between the two groups, the power of the test to be 80%, and the significance level (alpha) to be 5%, a sample size for each group of 22 persons was calculated (using EpiInfo, EpiCalc; see Table 1 for managers included).

The training program (educational intervention)
• Structure and objectives: the training program lasted 18 months.It contained five formal training courses, lasting five days each, with specific tasks and supervision by national facilitators in the interval periods of four months.
The training topics were: how to do needs assessment, local planning, monitoring and evaluation, drug management, risk approach, quality assurance, resource management, program management, information systems, and community participation.The principal objectives of the training programs were to: improve district health managers' knowledge and management skills; to strengthen creativity in responding flexibly to unexpected challenges; and to work as a team.The expected overall training outcome was better job performance.
The methodology used a participatory approach with different visualization techniques (work with cards on pin-boards), group work, didactic games, and socio-dramas for simulating real health problems.
• The replication workshops at district level: one important aim of the program was to replicate the contents and methodologies of the workshop program to the whole management team at district level.The replication workshops carried out by our trainees were a copy of the managers' training workshops and used the same training materials and evaluation forms as the "master" workshop.These are, however, not the subjects of this study.

Data processing and analysis
The data entry and processing was done using the EpiInfo program package.For the calculation of the risk ratio with their confidence intervals Statcalc of the EpiInfo program were used.The differences between before-and-after and intervention-control groups were calculated and the chi-squared test applied for statistical significance testing of the overall proportion of management techniques (or functions), which had not been used by health managers.Example from Figure 1: in a group of 33 health managers, each has to use 12 techniques, i.e. ideally these techniques have to be applied 396 times.However, in 13 cases they were not applied, resulting in a proportion of 3.3% of techniques which were not used.Additionally, the t-test was applied comparing mean number of practices adopted in the control and intervention groups.

Knowledge gain
The average knowledge gain in the workshops (as measured through knowledge tests) for the three intervention groups is presented in Figure 1.All three groups started at the same low level of knowledge (expressed as percentage of correct answers) and reached a similar high level of knowledge (in all three groups the dif- ferences in percentages of correct answers before and after the training were statistically significant; p < 0.001; the 95%CI were larger before the training [18-32; 13-29; 12-30] than afterwards [77-83; 80-88; 73-81]).

Training impact on job performance
• Outcome measurement: the main quantitative outcome indicators of the training program were the increased use of 12 key management techniques by the trainees (comparing Figure 2 with Figure 3).No specific weight was given to specific functions in order to avoid an arbitrary judgment about, which of these key functions are more important than others.In Figure 2 the differences between intervention and control groups in the baseline study are presented.None of them was statistically significant at the beginning of the training program.When comparing management performance in control groups "before and after" (averages in Figure 2 and Figure 3), the average management deficiencies did not change significantly (p > 0.05) although there was a certain fluctuation of individual management indicators in the three groups over time.
• Comparative impact measurement of the educational intervention: this was based on the 12 outcome variables indicating deficient use of 12 management techniques (Figure 3) and complemented by the qualitative study, which shed more light on the improvement in quality of management performance.From these 12 variables, the average proportions of management techniques, which have not been applied by district health managers, were calculated Figure 3).
The averages of the control group were divided by the averages of the intervention group in order to obtain the relative risk (RR, risk ratio).This was used as an estimate for the impact of the training program.The highest impact of the educational intervention was achieved in the Mexican group: the amount of managerial deficiencies was 8.3 times higher in the control group, compared to the intervention group.In Colombia, this factor was lower (3.6) as well as in El Salvador (2.4) (in all three cases p < 0.01) (Figure 4).The differences between Mexico and the other two countries was statistically significant (p < 0.01) but not between Colombia and El Salvador.The same significance level was obtained in the before-after comparison in the intervention groups; in the control groups no change occurred.
Additionally, the comparison was done by means of the t-test comparing mean number of management practices not carried out in the control and intervention groups.This takes into account that the unit of intervention was the person, and whether or not a specific practice is adopted is likely to be clustered within individuals.Also, in this case the differences between control and intervention groups were highly significant (p < 0.001).
The significant reduction in non-use of management techniques among trainees (intervention group) can be observed when comparing the black bars in Figures 2 and 3.The issue of "how adequately" these techniques have been applied by managers has been assessed through the visits to health districts in the qualitative follow-up study, where important changes in the intervention groups could be observed; staff interviews, discussions with management teams and document analyses reconfirmed the validity of interview answers.No change in the control groups compared to the baseline assessments was found.
• Other measures of program success: an indirect measure of the success of the program was the number of replication workshops to district health teams carried out by the workshop participants.Mexico: 147 replicated workshops for a total of 2,048 participants (on average 13.9 participants per workshop); Colombia: 62 replicated workshops for a total of 918 participants (14.8 participants per workshop) and El Salvador: no replicated workshops, as this was not yet a program element.practical work using operational research tools, exercises with a newly developed information system, field exercises on information synthesis and presentation and collection of insect vectors, participatory evaluation of each training workshop, and providing the right documents and literature.
Furthermore, the 18 months of the program provided enough time for trainers to better understand training needs and to adjust some training elements.This responds to Rogers' suggestion 22 (p.37) that "ideally, adult learners themselves should play an active role in training programs, identifying their training needs, participating in planning and implementation of their own training".(b) Strengthening variables associated with good performance: two variables which have been shown to be associated with good management performance 20 , have to do with training: "Knowledge of the local situation" and in a more general way "management training received".In the training program "knowledge of the local situation" was clearly strengthened through acquiring new tools to analyze the local situation and to summarize and present the information in an understandable way.Management training generally seems to have a  Likewise, the different tasks district health managers had to comply in the time between the training workshops, contributed to reinforcing the training; these included the production of a poster and document on district health problems; the design of a district health plan and the above mentioned replication workshops.The different achievements of the training program in the three countries seem to reflect these differences in reinforcement mechanisms: El Salvador was successful but less so than Mexico.The program was financed by foreign aid money and not directly integrated into the MoH structure.Participants had no time to do the replication workshops, and university lec- turers (instead of MoH staff ) did the supervision of tasks.In Colombia, supervision was less strict and the line management not as clear as in Mexico, but replicated workshops were conducted by most of the trainees and the supervision by MoH staff was important.Mexico was highly successful regarding training outcomes; all the reinforcement mechanisms described above were in place.
• Final appraisal of the training program and its outcome: the three expectations after training are: increase of knowledge; improvement of skills; and the practical application of both in future work.This has been achieved quite successfully in the three countries, but the longterm effects are unknown.It can be expected that, due to the high turnover rate of district health officers in El Salvador and some areas of Colombia, the positive effect of the training on district health management in these places is not sustainable.However, the recommendation will not be to discard such training but to institutionalize it in a systematic way.In addition, it is advocated that management training be integrated into quality assurance programs.The concepts used in performance management and quality assurance focusing on district health managers are similar; while activities in both can strengthen each other.The same participatory training methods can be applied with emphasis on teamwork and user involvement 23 such as were applied in the training programs in this research.
This experience with in-service management training confirms Stefanini's assumption 24 that management training cannot be separated from the work and activities being managed; that the different elements of management (such as human resources, finance or information) are best developed together and, that the organizational background has to be taken into account in management training.

Towards a refined methodology of measuring district health managers' job performance
• Limitations of the study: the lack of randomization of intervention and control groups as well as the difficulties with the control group in two countries means that the conclusions concerning the success of the educational intervention have to be taken with caution.However, this study is the first to measure the impact of management training in a quasi-experimental design and is in a position to provide considerably stronger evidence than has previously been available.• Outlook: in this research only the use or non-use of key management techniques by district health managers has been quantified, complemented by qualitative descriptive studies about managers' job performance.A more sophisticated way of performance measurement would be to quantify how appropriately these techniques are being applied and which impact they have on staff, people, and people's health.However, the literature has shown that this is difficult to establish and that the effect of appropriate management on people's health might be impossible to determine because many other powerful factors are in play, such as resources available, job imperatives limiting the flexibility of the manager's decisions, socioeconomic conditions of the target population, disease trends, and others 25 .Nevertheless, using the knowledge advanced by this research and complemented by more qualitative studies, it will be the challenge for the future to construct indicators on the appropriate use of key management tools, their direct effect on some outcome indicators, and how management training can modify them.

Figure 1 Average
Figure 1Average knowledge in the 3 intervention groups.

Figure 2 Percentage
Figure 2Percentage of managers in the intervention and control groups in the baseline study who do not carry out 12 key management functions in three countries.

Figure 3 Percentage
Figure 3Percentage of health managers in the intervention and control groups in the follow-up (post-intervention) study who do not carry out 12 key management functions in three countries.

Figure 4 Relative
Figure 4Relative risk (RR) of having management deficiencies when not participating in the educational intervention in three countries.

Table 1
Number of district health managers (DHM) from the Ministry of Health (MoH) and Social Security System (SSS)who received the training program (intervention group) or did not (control group) before and after the intervention.