A framework to assess management performance in district health systems : a qualitative and quantitative case study in Iran

The aim was to design a district health management performance framework for Iran’s healthcare system. The mixed-method study was conducted between September 2015 and May 2016 in Tabriz, Iran. In this study, the indicators of district health management performance were obtained by analyzing the 45 semi-structured surveys of experts in the public health system. Content validity of performance indicators which were generated in qualitative part were reviewed and confirmed based on content validity index (CVI). Also content validity ratio (CVR) was calculated using data acquired from a survey of 21 experts in quantitative part. The result of this study indicated that, initially, 81 indicators were considered in framework of district health management performance and, at the end, 53 indicators were validated and confirmed. These indicators were classified in 11 categories which include: human resources and organizational creativity, management and leadership, rules and ethics, planning and evaluation, district managing, health resources management and economics, community participation, quality improvement, research in health system, health information management, epidemiology and situation analysis. The designed framework model can be used to assess the district health management and facilitates performance improvement at the district level. Local Health Systems; Health Systems; Health Evaluation; Efficiency This article is published in Open Access under the Creative Commons Attribution license, which allows use, distribution, and reproduction in any medium, without restrictions, as long as the original work is correctly cited.


Introduction
District health systems are considered as an initial autonomous part of the national health system that combines primary health care with hospital care to provide integrated and complete services for the community 1,2 .Integrated district health systems have a substantial role in delivery of comprehensive health services 2 .However, it has been anticipated in the Alma-Ata Declaration that most developing countries would fail to achieve the level of a comprehensive health system would need to reform their management or governance practices 2 .
Iran's health sector Iran contains a variety of different players such as: Ministry of Health (MOH), Minister of Labor, Cooperative and Welfare, Minister of Defense, Ministry of Oil and many other public departments and private healthcare providers and medical insurers.So, the MOH is responsible for the population's health in the countryside and planning, monitoring, and supervision of public and private health sectors in Iran.MOH's executive responsibilities are delegated to university of medical sciences and health services at the provincial level.Medical universities are responsible for public health, health care provision in public facilities, and medical education.A nation-wide network developed to providing and monitoring healthcare acts at the district level (Figure 1) 3 .
The significant role of management and managers to construct effective health systems underlines the need for a comprehensive framework for district health management.This framework could Figure 1 Health system structure in Iran.
Cad. Saúde Pública 2018; 34(4):e00071717 include; planning, human resources development, and supervisory systems that improves health system management performance 4 .Literature shows the importance of these frameworks.For example, La Rue et al. 5 demonstrate that Leadership Development Programs increase health service coverage as well as the number of visits.Moreover, Dorgan et al. 6 found a positive association between management capabilities and health outcomes.
Assessing the performance of a health system to monitor and evaluate various aspects of the system and collecting data, review, refine and use this information are the most important factors in improving system efficiency and getting objective health system 7 .A district's health management performance framework is a conceptually structured tool to measure performance of policy makers, payers, providers, patients and population and analyze them to provide helpful information for decision-making 8 .Egger et al. 9 indicated that the lack of access to national and international guidelines and unclear responsibilities are the two most difficult barriers against achieving health system effectiveness.Therefore, different countries have adopted various approaches for management monitoring and evaluation programs using predetermined indicators.For instance, Gold Star Award in Uganda monitors the management performance of health facilities using 35 managerial standards in 47 districts 10 .District Health Barometer in South Africa uses predefined indicators and data to compare district health system performance at local and national levels 11 .
In Iran, district health system performance related to public healthcare is currently monitored through provincial health centers 12 and hospitals are evaluated by the national hospital accreditation system 13 .Although the hospital accreditation system is in progress to improve its standards and evaluation process, the current public healthcare monitoring program mostly focuses on inputs and health network structure instead of process and outcome indicators 14 .However, it seems that there is no specific and comprehensive package of indicators or framework to assess district health system management performance operationally in accordance with management framework 15 .So it seems that it is necessary to use a comprehensive set of indicators to assess district health system management performance and identify the strengths and weaknesses by health system authorities when planning and implementing corrective actions.Thus, this study tried to design a framework based on several indicators to assess district health management performance in Iran.

Design and setting
In this study, qualitative and quantitative methods were used for data collection.This study was conducted between September 2015 and April 2016 in Tabriz.Faculty members in health service management, unit chiefs in the northwest provinces of Iran Health Center and district health managers were the target population.The purposive sampling method was used to select expert managers.Forty-five participants took part in the first phase of study to develop indicators.In the second phase, 21 faculty members were surveyed to evaluate the indicators' content validity.In this phase, the participants were the district health managers and unit chiefs with experience in management evaluation.

Data collection
Semi-structured questionnaires were used to collect data from 45 participants and Table 1 shows participant characteristics.In addition, 21 questionnaires were filled out in quantitative part to validate and finalize the performance indicators of district health management.Study participants included those who were part of the health department or district health managers registered in health management training program and were trained regarding district health system management roles and capabilities.Semi-structured questionnaires were provided to study participants in last section of health management training program when they completed the training program 15 .

Interviewing and survey
As mentioned above, a semi-structured questionnaire was designed based on district health management capacity building framework and 11 main fields and topics was presented to them to facilitate and guide them to develop related indicators.The questionnaire guidelines consisted of the objectives of the study on performance indicators of district health managers about managerial competencies 15,16,17,18 .Each participant was to present SMART (Specific, Measureable, Attainable, Relevant and Time bonded) indicators based on their experiences to assess district health management performance of.Participants were fully free to present any indicators based on their own interest commensurate with the predefined categories of district health managers' management capabilities 15 .The analysis of questionnaires and indicator extraction was done simultaneously with collecting the questionnaires.Completing the questionnaires continued until saturation in participants' opinion.So, the latest questionnaires did not introduce any new indicator.
Analyzing the result of the qualitative part was used to extract performance indicators.Based on the findings of the qualitative part, the Validity Tool of District Management Performance Indicators was developed and its validity was examined and confirmed based on content validity index (CVI) and content validity ratio (CVR).Likert scale was used for each indicator.These indicators include the necessity (essential, useful, not essential, not useful), relevance (highly relevant, relevant, moderately relevant, not relevant), transparency (highly transparent, transparent, moderately transparent, not transparent), and simplicity (highly simple, simple, moderately simple, not simple).

• Phase 1: development of indicators
We conducted sequential data analysis, in which qualitative data is used to generate items for development of quantitative measures 19,20  pursued.Then, key themes were determined and were classified in logical format based on relationships.The research team explained the survey clearly to participants, highlighted the confidentiality research and got voluntarily informed consent.
Data collection continued until no new performance indicators were identified.After initial coding, data (indicators) were summarized as a structured thematic table.This step allowed researchers to easily have access to aggregated themes that extracted the participants' ideas regarding management performance indicators.Finally, clear consensus or non-consensus areas were highlighted, and relationships among indicators were explained in a logical framework.Discussion among the researchers created opportunity for more clarification, testing and revision of the findings.

• Phase 2: indicators content validity
Indicators which were generated in the first phase added to content validity questioners.Based on relevant, transparency and simplicity to calculate CVI and also each indicator assessed based on necessity to calculate CVR.CVI and CVR were calculated using the following formulas: CVI = N h /N CVR = (N e -N/2)/ (N/2) Where: N h = Number of panelists indicating "highly relevant or relevant" and "highly transparent or transparent" and "highly simple or simple"; N e = Number of panelists indicating "essential or useful"; N = Total number of panelists.
Also, the average CVI for final instrument was calculated based on Scale-Level CVI/Averaging Calculation Method (S-CVI/Ave).For this purpose the CVI of approved indicators was averaged to calculate S-CVI/Ave 21 .Microsoft Excel (Microsoft Corp., USA) was used for data analysis.

Ethical concerns
This study has been approved by the Tabriz University of Medical Sciences Research & Ethics Committee.In addition, the topic and the purpose of doing this study was explained to all participants and were entered into the study after getting oral and written consent.Finally, participants who refused to continue their participation in any step were excluded from study.

Participant characteristics in qualitative part
Overall, 45 experts filled out semi-structured questionnaires to identify district health management's performance indicators.Demographically, 86.7% of participants were male, 66.7% were district health managers, and 22.2% of them were head of unit in provincial health center.Also, 11.1% of participants had less than five years of experience in their current position while 55.6% had more than 20 years (Table 1).

Participant characteristics in quantitative part
In the quantitative part, 21 people participated.19% of participants were faculty members, 38.1% district health managers and 42.9% of them were head of unit in provincial health centers.The majority of participants were male (72.7%).Furthermore, 71.4% of the participants were medical doctors.Lastly, only 5.8% of participant had less than 5 years of experience and 42.9% of respondents had over 20 years of experience (Table 1).

District health management performance indicators
Based on the results of the qualitative part, 81 indicators were found to be related to district health management performance.Management performance indicators were classified into 11 categories, as following: human resources and organizational creativity, management and leadership, rules and ethics, planning and evaluation, district management, health resources management and economics, community participation, quality improvement, research in health system, health information management, epidemiology and situation analysis (Table 2).

Table 2
Management performance indicators for district health management assessment.

Row Category/Indicator CVR CVI Status
Human resources and organizational creativity   Out of 81 primary indicators, 68 of them that had been rated higher than 0.7 in CVI analysis were kept and 13 of them that had been rated less than 0.7 were excluded.On the other hand, 20 indicators were below CVR threshold, which was 0.42.Based on the cutoff point of 0.7 and 0.42 for CVI and CVR respectively (21 respondents who were rated CVR), 24 indicators were below CVI and CVR threshold and excluded from the final framework.In relation to indictors which were excluded from final list, in some cases participants believed that these indicators were out of district health management responsibilities and control such as: rate of achievements predicted budget, proportion of the population exposed to catastrophic costs, the existence of district data as electronic health records, or well presented in other accepted indicators as a comprehensive indicator.
Also, based on the result of the expert panel session, 4 indicators were merged with other indicators (n. 12 with n. 13, n. 56 with n. 54, n. 60 with n. 55 and n. 76 with n. 79) (Table 2).
Finally, the 53 most relevant indicators remained.The average CVI (S-CVI/Ave) of approved indicators was 0.84.Also, study findings indicate based on participants' viewpoints epidemiology and situation analysis reached highest average CVI (S-CVI/Ave = 0.90) and research in health system reached least average CVI (S-CVI/Ave = 0.78) (Table 3).

Discussion
According to the results, 53 indicators were included in district health management performance framework.The average CVI (S-CVI/Ave) of the final framework was 0.84.The governing role of the district health system in overseeing service providers and monitoring their performance were the most important part of management performance framework.
Siddiqi et al. 22 provided a framework to assess health systems performance in developing countries with 10 principles including: strategic vision, participation and consensus orientation, rule of law, transparency, responsiveness, equity and inclusiveness, effectiveness and efficiency, accountability, intelligence and information and ethics.Brinkerhoff et al. 23 indicated that responsiveness, leadership, customer voice, accountability, transparency, evidence-based practice, efficiency and effectiveness as governing outcome indicators in district health systems.Also, in some other countries, the World Health Organization's (WHO) building blocks were used to assess district health management and related interventions 24 .Whereas, it should be kept in mind that these frameworks weren't developed as specific tools to assess managerial performance at district levels.
Based on this analysis, indicators that were related to motivation and human resources developments as well as reward and performance management systems were classified under human resources management category.Dieleman et al. 25   study, Kennedy 27 defined HRM performance based on employees' incentives and rewards, active participation in continuing education and comprehensive feedback systems.In this study, human resource management indicators were classified in a different category than resource management.However, in literature, other studies have classified them in the same category.Also, from the perspective of study participants, approved indicators in this category were able to represent district health system management performance in motivating and directing human resources in health network to achieve its objectives.
Resource management includes budget management, costs and financial resources, equipment and facility management.Dadgar et al. 28 included three indicators as equipment, facility management, and financial resources in the resource management category.Kennedy 27 identified unit cost as a single indicator for resource management.Accordingly, based on the current study's results, a comprehensive range of indicators can be used as a useful tool to assess Resource management.It appears, due to legal and regulatory limitations, district health managers were not completely free and autonomous to change system inputs and structure to improve efficiency and effectiveness.So, some of predefined indicators in the current study were rejected by participants because district health authorities do not have the power to make changes.
Based on our findings, the indicators of district health management compiled for empowerment of managers.In this respect, some studies have confirmed that managers' performance must be assessed based on their skills and capabilities.Dadgar et al. 28 found that manager participation in programs, employees' performance assessment and rewarding systems indicates managers' performance.Kennedy 27 demonstrated that work experience, competency and having a comprehensive approach are the most important indicators of managers' performance.
Systematic processes for human resources management, complaints management and customer satisfaction have been considered as rules and ethics management indicators.Dadgar et al. 28 showed that employee and customer satisfaction and their complaints must be categorized differently.It seems that the existence of lobbying and politics in managerial decisions at district levels restrict the strict implementation of rules based on rational decisions 29 .As a result, some of the indicators, such as compliance with employing position requirement that were out of district health managers' controls were removed from the final list.
The existence and implementation of action plans and also coordination with relevant organizations were considered as planning and evaluation indicators of.The study performed by Dadgar et al. 28 pointed out that the existence of strategic and operational planning and stakeholder participation in program development are planning indicators.Fang et al. 30 found that planning is an important Cad.Saúde Pública 2018; 34(4):e00071717 dimension in management performance framework.Though, it should be noted that development and implementation of strategic and operational planning in district health systems from the perspective of participants can represent the existence of a monitoring and evaluation program in the health network.
According to our results, supporting the quality improvement programs and enriching the infrastructural capacities to improve the quality of services were quality management indicators.Also, research in the health system area is closely related to quality improvements.In addition, based on our findings, it is necessary that employees participate in problem solving training programs.Moreover, data collection and data analysis on the performance indicators and health services indicators, in addition to feedback and evidence based planning, were other categories in the district health system management performance framework.Dadgar et al. 28 indicated that clinical governance, quality improvement, data collection and information management were four important components of their framework.Brinkerhoff et al. 23 concluded that high quality service provisions and customer satisfaction are important indicators of district health system management performance.Based on WHO's building blocks, quality and safety are the intermediate goals of any health system.Kennedy 27 argued that access to the latest information technology, information utilization and information sharing must be considered as management performance indicators.
By participating in health services prioritization of, the community will trust the health system, the health system's responsiveness of will improve and people will collaborate more to achieve an efficient and cost effective system 31,32 .Community participation promotes healthy behaviors, service provision, resource mobilization and community empowerment 33 .In this regard, advocacy and inter-sectoral collaboration are two effective strategies for health promotion and health system development 34 that were an important category in management framework.Therefore, inter-sectoral collaboration and community contribution had the same score as the management effectiveness indicators and managerial capabilities and competencies in health systems 35,36,37 .
In this study, the management performance indicators were extracted using a qualitative and quantitative study of managers from the district health system, faculty members and other participants.However, identified indicators weren't weighted and ordered.On the other hand, the real world applicability of a designed framework must be evaluated in health systems to find out its weaknesses and limitations.Also, a small sample size in the quantitative phase makes it necessity to use real field data to assess framework validity.These indicators must be practical and applicable for district health system.However, we identified the comprehensive collection of indicators and the results of this study can be useful for monitoring and improving district health system management.The proposed framework as a comprehensive assessment tool can be used by national and provincial health authorities to monitor and direct district health systems.District health authorities can also use the current framework as a self-assessment tool and performance improvement guide to manage their own programs and fully respond to their duties.For this purpose, the first step is developing approved indicators as a check list format and cut off points must be determined for specific indicators such as unit costs, catastrophic payment, health insurance coverage rate and healthcare provider per population.

Conclusion
A district health system management performance assessment framework was developed according to district health managers' capacity building program and covers all dimensions of district health management, such as: human and capital management, leadership, governance, planning and evaluation, and efficiency and effectiveness.Considering the relatively high validity and reliability of acquired indicators and the absence of a comprehensive framework to assess the performance of district health managers, it seems that this framework can be helpful to assess the performance of district health management and create performance improvement programs at the district level.It has been recommended that provincial health system managers provide regular monitoring programs and feedback to district health managers using this framework.Also, health managers can use this framework to control and manage district health systems.
Cad. Saúde Pública 2018; 34(4):e00071717 Contributors J. S. Tabrizi carried out proposal drafting, participated in data collection and helped drafting the manuscript.K. Gholipour carried out proposal drafting, developed the study design, participated in data collection, performed analysis and helped drafting the manuscript.S. Iezadi participated in data collection, performed analysis and helped drafting the manuscript.M. Farahbakhsh participated in study design and coordination, and helped drafting the manuscript.A. Ghiasi developed the study design, provided coordination and helped drafting the manuscript.

Table
Demographic characteristics of participants in qualitative and quantitative phases.
and, in our case, to generate performance indicators.Content analysis and comparison of performance indicators which were specified by the participants were

Table 2 (continued)
26ouped continuing education, supervision, incentives payment, legal and regulatory compliance, improving quality of care and mixed targets under human resources development interventions.In another study, Liu et al.26categorized human resource management (HRM) indicators into three areas; appropriate staff numbers, appropriate skills and experiences, and providing appropriate productivity to organizational performance.In another Cad. Saúde Pública 2018; 34(4):e00071717

Table 3
Average content validity index (S-CVI/Ave) of district management performance categories.