Decision criteria for resource allocation: an analysis of CONITEC oncology reports

In health technology assessment (HTA), decision criteria are considered relevant to support the complex deliberative process that requires simultaneous consideration of multiple factors. The aim was to identify and analyze the decision criteria that have been used by the National Health Technology Assessment Commission (CONITEC) when recommending the incorporation of technologies for the treatment of cancer. Descriptive study, based on reports from CONITEC, between 2012 and 2018, on oncology technologies. The data were collected in a specific extraction form and analyzed using descriptive statistics. 39 reports were analyzed, 15 of them did not present any explicit decision criteria. Medicines were the most frequently evaluated type of technology. The most frequent types of cancers were: breast cancer, head and neck cancer, colorectal cancer, non-Hodgkin’s lymphoma and lung cancer. The most frequently considered criteria were: financial impact and effectiveness. The study identified the decision criteria that have been most used in the area of oncology, however, the lack of transparency in relation to the weight of these criteria makes it difficult to understand their influence on the result of the decisions taken.


Introduction
According to estimates by the World Health Organization (WHO) in 2019, cancer is among the four leading causes of death in high-and middle-income countries, constituting the most important barrier to raising life expectancy worldwide 1,2 . The prevalence of cancer in developing countries has reached levels close to those recorded in developed countries, and in Brazil, for each year of the 2020-2022 triennium, there have been an estimated 625,000 new cases of cancer 3 .
To confront this challenge, in 2005, the Ministry of Health proposed to institute the National Policy on Oncology Care through an ordinance, covering promotion, prevention, diagnosis, treatment, rehabilitation, and palliative care, to be implemented throughout the country 4 . However, it is important to understand the budget impact of this policy, as well as to optimize the allocation of resources and to establish priorities for the approach to cancer within the scope of the Unified Health System (SUS).
In this regard, the incorporation of health technologies is a challenging activity that could contribute to the sustainability of public policies, but which requires careful consideration of multiple factors, such as: disease burden, clinical benefit, safety, level of innovation, quality of clinical evidence, cost-effectiveness, budgetary impact, and other relevant sources of evidence 5, 6 . By observing these factors in the context of SUS, resources are being allocated in an efficient way based on the principles of health technology assessment (HTA). To this end, decision criteria in HTA processes can be defined as factors that are considered relevant to a decision-making process 7 and that, in principle, should support the complex deliberative process that requires simultaneous consideration of multiple factors 8 .
Decisions aimed at incorporating new technologies in the field of oncology have become increasingly complex and multifaceted due to rapid advances in research and the heterogeneity of available scientific evidence. 9 Some oncology treatment decisions are less complex, with clear evidence to support a better approach, while others may depend on considerations of conflicting factors and the absence of shared opinions about the role of evidence and value judgments in health 5, 9 .
The technologies used in the field of oncology in Brazil must be evaluated through an HTA process by the National Health Technology Assessment Commission (CONITEC) to be incorporated for reimbursement by SUS 10 . Based on the National Health Technology Management Policy (PNGTS) 11 , CONITEC considers eight relevant criteria for the incorporation of technologies into SUS: safety, efficacy, effectiveness, efficiency, and budgetary, ethical, social, and environmental impacts of the technology in question. The criteria used by CONITEC when preparing recommendation reports on the incorporation of technologies for SUS are part of a complex decision-making process that includes: a) a comprehensive, systematic review of the scientific literature; b) a critical assessment of the best available evidence; c) consideration of the population's health needs and health policy priorities; d) market characterization and choice of comparable therapeutic alternatives; e) an evaluation of the logistics for the implementation of the new technology, according to SUS care protocols; and f) an evaluation of the technology's effectiveness, safety, cost-effectiveness, and budgetary impact, according to studies presented by the applicant for the incorporation of the technology in question, in addition to other criteria based on the PNGTS (from the perspective of SUS) 10,11 .
Previous studies carried out in our country have shown that the nature and type of evidence used in CONITEC recommendation reports are unclear 12 and that economic and clinical factors are those most frequently used to guide CO-NITEC recommendations 10 . However, it is unknown how these factors may have influenced the recommendations for incorporating technologies related to the approach to cancer in Brazil.
Thus, this study aims to identify and analyze the decision-making criteria that have been used by CONITEC in recommending the incorporation of technologies for the treatment of cancer within the scope of SUS, based on an analysis of CONITEC's recommendation reports from 2012 to 2018.

Methods
This is a descriptive study, based on official open access reports from CONITEC that deal with oncology technologies. CONITEC recommendation reports were chosen as empirical material for analysis as, in addition to being the final product of the entire recommendation flow of a technology for incorporation into SUS, they illustrate the use of scientific evidence in this process and the criteria prioritized by decision-makers.
Data were read and extracted from all recommendation reports related to the approach to cancer, publicly available on the CONITEC web-site, published between July 1, 2012, and July 1, 2018. This period was selected because it included a relatively broad period of six years, covering two administrations of the federal government A standardized form was prepared prior to data extraction to collect variables of interest and a pilot phase was conducted aimed at extracting data from three eligible reports to standardize the terms to be used, train the reviewers, and validate the form. Two independent reviewers then extracted data from the reports using the standardized form. Disagreements between reviewers were resolved through discussion and consensus.
For the purpose of this study, information was initially extracted from the reports regarding the type of technology, description of the characteristics and current use of the technology, year of the report, and type of diagnosis, based on previous scientific publications 12,13 . The diseases targeted by the technologies evaluated in the reports were classified according to the sections of the tenth revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10). Only oncology-related reports were included.
Secondly, the reports were evaluated in order to extract information regarding the inclusion of studies and considerations that inform decision-making criteria, based on the proposal suggested by Merlin et al.: 14 1) safety assessment, 2) evaluation of effectiveness, 3) analysis of cost-effectiveness, 4) acquisition costs and budgetary impact, and 5) organizational considerations.
The extracted data were then evaluated for the presence of information or studies described in the body of the report with regard to each of the eight criteria for incorporating technologies contained in the PNGTS: safety, effectiveness, and efficiency, as well as budgetary, ethical, social, and environmental impacts of the technology in question 11 . Afterward, the weight of these criteria was evaluated in terms of the presence of comments or justifications related to each of them in the final recommendations of the reports (Chart 1). Thus, by examining the content of the final recommendations, the reviewers identified references to studies included by the applicant in the report, along with information from these studies or mention of the decision-making criteria (or even definitions of these criteria), which could characterize the relevance of these contents for the recommendations made, according to the corresponding decision-making criteria. Likewise, the reports with changes in their final recommendations after public consultation by CONITEC were identified, and a textual analysis of the final recommendations sought to identify information regarding the decision-making criteria that could justify changing the technology recommendation.
The information from each report was collected using an extraction form developed for this purpose and analyzed in a descriptive way. Frequencies and percentages were used for data analysis, using Microsoft Excel spreadsheets.

Results
Thirty-nine CONITEC recommendation reports were included regarding the incorporation of technologies for approaching cancer in SUS. The number of reports corresponded to 13.8% of the total reports by CONITEC in 2012, 16 Table 1 shows the types of technologies evaluated between 2012 and 2018. Medicines were the most frequently evaluated type of technology in the period, with the highest number of reports being produced in 2013 and 2014 (9 reports in each year). Table 2 presents the types of cancer that were most often considered in the CONITEC recommendation reports, highlighting the most frequent types: breast cancer (8 reports), head and neck cancer (4 reports), colorectal cancer (4 reports), non-Hodgkin lymphoma (4 reports), and lung cancer (3 reports).
Among the thirty-nine recommendation reports that were analyzed, fifteen (36.46%) did not present any information that reported an explicit decision-making criterion or based on the classifications considered in the study. The other twenty-four reports presented criteria informed by evidence as described in the classification by Merlin et al., 14 and in several cases, the report presented more than one criterion. Table  3 provides a description of the decision-making criteria presented in the reports, based on guidelines considered by CONITEC and in the classification of Merlin et al. 14 Of these, 43.59% presented information regarding the criterion of effectiveness/efficacy and 38.46% related to the criteria of acquisition costs and budgetary impact, which were those most frequently informed with evidence submitted by the applicant for the incorporation. Table 4 presents the frequency with which the PNGTS decision-making criteria are reported and considered in the body and in the final recommendations of the CONITEC reports, the most frequent in the body of the report being budget impact (in 87.18% of the reports) and effectiveness/efficacy (in 74.36% of the reports). Regarding the final recommendation, the most frequent criteria were effectiveness/efficacy (in 43.59% of the reports) and budgetary impact (in 33.33% of the reports). Table 5 presents the decision-making criteria considered in the changes made in the CONITEC recommendation after public consultation. There were three technology cases that resulted in a change in CONITEC's position after public consultation, referring to the following drugs: gefitinib, erlotinib, and pertuzumab. Additional evidence was presented during the public consultation in the field of technology effectiveness/efficacy, budgetary impact, information on efficiency (based on cost-effectiveness analysis), which motivated the change in recommendation. The effectiveness/efficacy criterion was the only one that was considered in all changes made in the final recommendation after public consultation.

Discussion
This study aimed to identify and analyze decision-making criteria that have been used by CONITEC in the processes of incorporating technologies in the approach to neoplasms. The analysis of the recommendations related to cancer made by the commission from 2012 to 2018 showed that acquisition costs and budgetary impact were the criterion that most frequently appeared in the body of the reports whereas the effectiveness/efficacy criterion appeared more frequently in the final recommendation section. However, when there is a change in the commission's initial recommendation based on public consultations and the PNGTS, the effectiveness/ efficacy and budgetary impact criteria are those most frequently considered. Other criteria appearing in the PNGTS do not appear in the re-  ports, and it was not possible to understand how these criteria are added or reconsidered after public consultation.
Regardless of the relevance of the decision-making criteria, it was initially possible to observe some distinction between the types of cancer most often evaluated in the reports and their estimated incidence. In Brazil, the estimate for each year of the 2020-2022 triennium indicates there will be 625,000 new cases (450,000, excluding non-melanoma skin cancer cases), with non-melanoma skin cancer being the most frequent (177,000), followed by breast and prostate (66,000 each), colon and rectum (41,000), lung (30,000), and stomach (21,000) cancers 3 . By contrast, Table 2 shows three of these types of neoplasms being among the most frequently evaluated. In descending order, they are breast, head and neck, colorectal, non-Hodgkin's lymphoma, and lung.
In a previous study, Elias et al. showed that CONITEC typically prioritizes clinical criteria (for example, efficacy) and that recommendations favorable to the incorporation of the required technologies are often guided by clinical or therapeutic impact criteria (gain in efficacy is mentioned in 71% of the evaluated reports) as compared to other criteria. 10 Regarding economic considerations, the budgetary impact criterion (mentioned in 38% of the reports) appears most frequently in the recommendations, although in a much lower percentage than in the therapeutic impact criterion 10 . Table 3. Description of decision-making criteria used in recommendations of CONITEC reports related to the approach to cancer, 2012-2018.

Effectiveness/ efficacy
Effectiveness: Probability that individuals from a defined population will benefit from the application of a health technology targeted at a given problem under real conditions of use. Efficacy: Probability that individuals in a defined population will benefit from the application of a health technology targeted at a given problem under controlled conditions of use.  In another study, Yuba et al. 12 also analyzed CONITEC's recommendation reports and concluded that the characteristics of the evidence that were considered mandatory for the decisions were very different, indicating problems in the decision-making process 12 . One of the most important findings of that study was CONITEC's lack of adherence to its own internal regulations in terms of evidence to be considered and, therefore, of the decision-making criteria that guided the final recommendations 12 .
Given the progressive increase in healthcare costs, explicit approaches to the prioritization and allocation of resources have become critical and illustrate how decision-making criteria is a key element in ensuring transparency and, therefore, legitimacy of the decisions being made 8 . In accordance with the findings of Elias et al., this study reveals that the effectiveness/efficacy criterion is the one most often considered in CO-NITEC's final recommendations 10 , although information on the budgetary impact criterion is more frequent found in the body of the report, and other criteria are not considered or reported, despite being mentioned in the PNGTS 11 . Regarding the budgetary impact criterion, of the 101 reports evaluated by Yuba et al., only 19.8% included a complete economic assessment, despite the commission's internal regulations establishing them as mandatory for all demands for incorporating technologies 12 .
Considering the clear difference in the scientific evidence, depending on the type of demand for technology incorporation (internal or exter-nal to SUS) 12 , this study's findings suggest a need for greater transparency in CONITEC's internal decision-making processes, in addition to showing an overestimation of certain criteria in the final recommendation (clinical, in particular) in relation to other PNGTS criteria. In this light, it is possible to conclude that clinical criteria (such as efficacy and effectiveness) actually end up exerting a greater influence on the commission's final recommendations, although economic criteria (such as budgetary impact and cost-effectiveness) are frequently cited throughout the reports.
In a literature review of studies on health-related decision-making, Guindo et al. identified a wide variety that have often been considered 8 . In the forty articles, the five most frequently mentioned criteria were in descending order: equity/fairness, efficacy/effectiveness, stakeholder interests and pressures, cost-effectiveness, and strength of evidence. In another study, the results from a literature review showed that 25 of the 40 studies included in that search dealt with "health outcomes" as the main group of criteria in defining priorities for health technology assessment, and that the higher frequency of "health effects/ benefits" criteria (in eight of the studies) is in line with the findings of this study 15 . Thus, the relevance of clinical criterion (effectiveness/efficacy) seems to be corroborated by studies conducted in other contexts, according to the literature reviews. However, the equity/justice criterion (the one most frequently represented in studies of different nationalities, according to the study by Guindo et al.) 8 , which may even be associated with the ethical impacts of health decisions, is not explicitly presented in the revised CONITEC reports. Nonetheless, the non-inclusion of this criterion in the reports may be due to a difficulty in considering equity in a pragmatic way; which points to the need for systematic approaches that facilitate the operationalization of this criterion in the decision-making process.
To assist in the process of including criteria that are difficult to quantify, there is a wide set of techniques that can help make the decision-making process more systematic and transparent, often called multi-criteria decision analysis (MCDA) [16][17][18] . The use of structured methodological approaches, such as MCDA, could be an option to facilitate decision-making processes and assist in preparing recommendations within the scope of CONITEC. The MCDA is comprised of a set of methods to support decision-making that explicitly takes into account multiple criteria, allowing different stakeholders to participate 16 . The basic aspects of the method involve: 1) the selection of alternatives to be evaluated, 2) the choice of criteria through which the alternatives will be compared, 3) the measurement of the value attributed to the performance of the technologies in each of the criteria, and 4) the measurement of the weight of the criteria in order to reflect the relative importance of each one 16,17 .
The use of an MCDA methodology could allow the assessment of stakeholder preferences (in relation to decision-making criteria), help ensure that they are more systematically taken into account in the decisions, and estimate the overall value of each one, based on monitoring the stages of a discussion leading up to collective decisions 8 . In HTA processes, one of MCDA's greatest contributions is to increase transparency in the incorporation of technology in a more participatory and legitimate way.
When it comes to oncology, a previous systematic review of current practices and applications of MCDA showed cancer as being the most researched type of disease 14 . In another review dedicated to MCDA applications in oncology, although the eight studies reviewed were focused on clinical decision-making (six for cancer screening), the authors recognized the importance of the approach to assessing stakeholder preferences and developing clinical guidelines 9 . Furthermore, other studies reviewing the scientific literature have highlighted the importance of MCDA methods in prioritizing interventions for coverage or reimbursement, supporting decisions in health technology assessment [19][20][21][22][23][24] .
Therefore, when preparing an MCDA, it is important to consider that the selection and structuring of criteria must follow good practices that justify their selection and adequate reporting in published studies, as recommended by the Professional Society for Health Economics and Outcomes Research (ISPOR) 17 . The ISPOR recommendations highlight the importance of investigating reports that guide decision-making, such as those prepared by HTA agencies, e.g., the National Institute for Health and Care Excellence (NICE) in the United Kingdom, 25 and, in the case of Brazil, CONITEC's role in preparing HTA reports.
As with MCDA, other prioritization and decision support processes could be considered, such as portfolio analyses, which address the question of how to combine interventions within or across programs to maximize a given objective, and approaches based on program budgeting and marginal analysis that aim to maximize defined objectives by choosing from a range of interventions, keeping budget constraints in mind 26 . Even so, these approaches use various criteria, such as those identified and analyzed in this study, in an attempt to reflect the reality that decision-makers typically face when having to deal with a set of objectives while evaluating the value of a new technology 27 .
Although none of these approaches is used to support decisions on the incorporation of health technologies, identifying and evaluating the frequency of decision-making criteria in recommendation reports from an HTA commission, such as CONITEC, already allows a structure or framework of value to be used as a way to assist the commission's deliberative work. In this regard, the implementation of various frameworks of value has been proposed by international bodies, such as the American Society of Clinical Oncology (ASCO), the American Heart Association (AHA)/American College of Cardiology (ACC), the Institute for Clinical and Economic Review (ICER), the Memorial Sloan Kettering Cancer Center (MSKCC), and the National Comprehensive Cancer Center (NCCC) 28 . The main objective of these frameworks is to inform the decisions of different stakeholders, based on robust clinical evidence and shared value judgments 27 .
Therefore, the main contributions of this study are, first, to present the decision-making criteria that have been considered most frequently by CONITEC when evaluating technologies for the approach to cancer and, second, to show that, as the reports do not clearly present the weights of these criteria and the value assigned to technology performance; it is difficult to understand the influence these criteria have on the outcome of the decisions.
However, these results need to be viewed with care insofar as qualitative analyses were not conducted with the participants of the aforementioned decision-making processes for a better understanding of how the decision-making criteria were considered. Another limitation to be considered is that it is not possible to ascertain which other criteria were not considered or why they were not reported from the analysis of reports. A third limitation refers to the fact that the evaluation was based on reports that may not include the criteria that really weighed on the decision, meaning the analysis is limited to recorded information. Another thing to understand is that not all public consultations were considered, which could provide additional information about the deliberative processes. Finally, as reports from mid-2018 onwards were not evaluated, it is not possible to verify changes in CONITEC's decision-making criteria in recent years.
Ultimately, it is important to remember that HTA is a socio-technical process by nature and that its recommendations need to reflect its social aspects (who participates, how, and in what way) and technical aspects (which methods are used) in a systematic and transparent way. Despite the challenges imposed by this process, the advances made in CONITEC's performance in our country is undeniable, considering there has been a significant rise in the production of literature in the HTA area in Brazil since 2008, as well as in the development of training courses in systematic reviews, technical-scientific reports, and promulgation guidelines. 12 In terms of the challenges posed by innovations in the field of oncology, demystifying the decision-making criteria and how they are considered is essential to ensuring the legitimacy of decisions to incorporate new health technologies.

Conclusion
This study identified that the decision criteria that have been more informed in the CONITEC recommendation reports for the incorporation of technologies for the treatment of cancer, from 2012 to 2018, were: first, budgetary impacts, followed by effectiveness /efficiency. Yet, in the final recommendations, effectiveness/efficacy was the most frequently considered.