Development of an indicator matrix for the quality of care for older adults in emergenciesABSTRACT
Objective: To develop an indicator matrix to assess the quality of care provided to older adults in Emergency Care Units.
Method: Applied methodological study, conducted in three stages, between September 2023 and March 2024: Bibliographic survey and selection of structuring foundations; Development of the analysis and judgment matrix; Content validation by a panel of experts (16), using the Delphi technique, calculating the Agreement Rate and the Content Validity Index.
Results: At the end of the two rounds, the matrix showed an overall agreement of 0.94. As for the Content Validity Index, the mean values were 0.95 in the first round and 0.93 in the second round. After adjustments recommended by the expert judges and the reallocation of 2 indicators between the process and outcome dimensions, the matrix comprised 16 items, distributed as follows: structure (n=7), process (n=3), and outcome (n=6).
Conclusion: The matrix was conceptually validated to measure the quality of care for older adults in Emergency Care Units and to capture specific aspects across the three dimensions addressed. It proved to be a technology capable of supporting managers and professionals in decision-making, enabling continuous, sustainable improvement in the quality of care for the older population.
Descriptors:
Older adults; Emergency medical services; Quality of health care; Health care quality indicators; Validation study
RESUMO
Objetivo: Desenvolver uma matriz de indicadores para avaliar a qualidade do atendimento à pessoa idosa nas Unidades de Pronto Atendimento.
Método: Estudo metodológico aplicado, conduzido em três etapas, entre setembro de 2023 e março de 2024: Levantamento bibliográfico e seleção dos fundamentos estruturantes; Desenvolvimento da matriz de análise e julgamento; Validação de conteúdo por um painel de juízes (16), por meio da técnica Delphi, calculando-se a Taxa de Concordância e o Índice de Validade de Conteúdo.
Resultados: Ao final das duas rodadas, a matriz apresentou uma concordância global de 0,94. Quanto ao Índice de Validade de Conteúdo, os valores médios foram de 0,95 na primeira e de 0,93 na segunda rodadas. Após ajustes recomendados pelos juízes e a realocação de dois indicadores entre as dimensões processo e resultado, a matriz contabilizou 16 itens, distribuídos em: estrutura (n=7), processo (n=3) e resultado (n=6).
Conclusão: A matriz foi validada conceitualmente para mensurar a qualidade do atendimento à pessoa idosa nas Unidades de Pronto Atendimento, bem como para captar aspectos específicos nas três dimensões abordadas. Observa-se que a matriz é capaz de subsidiar gestores e profissionais no processo de tomada de decisões, possibilitando a melhoria contínua e sustentável da qualidade da atenção à população idosa.
Descritores:
Idoso; Serviços médicos de emergência; Qualidade dos serviços de saúde; Indicadores de Qualidade em Assistência à Saúde; Estudo de validação
RESUMEN
Objetivo: Desarrollar una matriz de indicadores para evaluar la calidad de la atención a la persona mayor en las Unidades de Atención de Urgencia.
Método: Estudio metodológico aplicado, realizado en tres etapas, entre septiembre de 2023 y marzo de 2024: Levantamiento bibliográfico y selección de fundamentos estructurantes; Elaboración de la matriz de análisis y juzgamiento; Validación de contenido por un panel de jueces (16), utilizando la técnica Delphi, calculando la Tasa de Concordancia y el Índice de Validez de Contenido.
Resultados: Al final de las dos rondas, la matriz mostró una concordancia general de 0,94. En cuanto al Índice de Validez de Contenido, los valores promedio fueron de 0,95 en la primera ronda y de 0,93 en la segunda. Tras los ajustes recomendados por los jueces y la reasignación de dos indicadores entre las dimensiones de proceso y de resultado, la matriz contabilizó 16 ítems, distribuidos de la siguiente manera: estructura (n=7), proceso (n=3) y resultado (n=6).
Conclusión: La matriz se validó conceptualmente para medir la calidad de la atención a las personas mayores en unidades de urgencias, así como para captar aspectos específicos de las tres dimensiones abordadas. Se observa que la matriz puede apoyar a gestores y profesionales en el proceso de toma de decisiones, lo que posibilita la mejora continua y sostenible de la calidad de la atención a la población mayor.
Descriptores:
Persona mayor; Servicios médicos de urgencia; Calidad de los servicios de salud; Indicadores de calidad de la atención en salud; Estudio de validación
INTRODUCTION
The increase in life expectancy of the population has been observed on a global scale. A demographic inversion is evident due to the continuous decline in birth rates, illustrated by an age pyramid with a narrowed base and a widened top, assuming a barrel-shaped form1. According to the World Health Organization (WHO), between 2015 and 2050, the world population aged 60 or over is expected to practically double, going from about 12% to approximately 22%2.
Although the growth of this population group is more pronounced in developed countries, Brazil presents a similar phenomenon, with an exponential increase of 29.9 million people3. Demographic changes in the national scenario project a proportion of older adults (60 years or older) of around 37.8% by the year 20703.
Given the above, there is an increase in the flow of older adults in emergency services, whose search for care occurs due to the exacerbation of their pathological history, aggravated by the low problem-solving capacity of Primary Health Care (PHC) in managing chronic conditions4. Studies reveal that aging, associated with the high prevalence of Noncommunicable Chronic Diseases (NCDs) and other conditions that lead to declines during senescence, results in the decompensation of preexisting conditions, thereby intensifying the demand for emergency services5.
In this perspective, the 24-hour Emergency Care Unit (UPA 24h) emerges as an important entry point for emergency care, integrating the Emergency Care Network (ECN). These units have physical and technological infrastructure of intermediate complexity, as well as specialized multidisciplinary teams, acting as support for acute PHC cases and stabilizing more critical cases for subsequent referral to the hospital network through medical regulation6.
Although the UPA 24h was conceived as a strategy to reduce overcrowding at hospital emergency departments and to optimize integration with the healthcare network, recent studies indicate that these units are overcrowded, with disorganized population access, and face regulatory difficulties, particularly in referring older adults to the hospital network7-8. It can therefore be inferred that this population is more susceptible to weaknesses in care, which are exacerbated by the complexity of their clinical profile, whose management requires a qualified and specialized approach, especially in urgent and emergency contexts7-9.
These limitations reinforce the need to assess the quality of care provided in UPA 24h units, particularly regarding care for older adults, through care indicators. The adoption of Donabedian’s Triad (structure, process, and outcome)10 allows for a systematic and measurable evaluation, translating resources, practices and outcomes into concrete indicators. This approach was recently used in a study evaluating quality in PHC, urgent/emergency, and hospital services within the context of the Unified Health System (Sistema Único de Saúde - SUS)7. In this context, the regulation and monitoring of public health services recommend the application of quality indicators for auditing and control of care11, which legitimizes the theoretical and methodological option adopted.
In the same direction, health evaluation identifies the weaknesses and potential of services, guiding strategies for continuous and sustainable improvement11-12. Although the Ministry of Health (MH) monitors the performance of the ECN through evaluative tools, recent studies point to a fragmented process, centered on the isolated evaluation of services12-13. Furthermore, this study is justified by the need and importance of proposing an integrated evaluation of the quality of care for older adults in UPA 24h units, considering indicators distributed across the three Donabedian dimensions.
Thus, the developed matrix is configured as an innovative evaluative technology for improving healthcare through the quality attribute. Within the scope of this research, the following questions emerge: is the developed matrix capable of evaluating the quality of care for older adults in UPA 24h units? The study is based on the hypotheses that (i) the instrument presents consistency in content validation and (ii) its indicators measure aspects embedded in the structure, process, and outcome dimensions10.
The relevance of this study lies in proposing an evaluative technology that allows measuring the quality of care for the elderly in UPA 24h, contributing to the improvement of management and the strengthening of the ECN. Thus, the objective of this research was to develop an indicator matrix to assess the quality of care provided to older adults in UPA 24h units.
METHOD
Type of study
This is an applied methodological study aimed at developing an indicator matrix to assess the quality of care provided to older adults in UPA 24h. The study was conducted from September 2023 to March 2024, in three stages: Literature review and selection of structuring references; Development of the evaluative indicator and judgment matrix; Content validation (CV) of the matrix by a panel of experts using the Delphi technique14-15, calculating the Agreement Rate (AR) and the Content Validity Index (CVI).
Applied methodological studies are characterized by the development, adaptation, validation, or improvement of instruments or technologies intended to measure constructs, evaluate processes, and qualify healthcare practices. Such studies have been increasingly used in the field of nursing and health evaluation16-17.
Content validation, in turn, constitutes a stage of the methodological study, as it allows verification of the degree of representativeness, relevance, and clarity of the indicators in relation to the construct intended to be measured. This process is conducted by experts in the field, who assess whether the items adequately and sufficiently reflect the proposed theoretical dimensions (18-19.
In this study, the evaluative matrix was conceived as an evaluative technology of a conceptual and normative nature, structured based on Donabedian’s triad10, integrating scientific evidence, guidelines, and expert judgment. Thus, the adopted design follows recommendations for methodological studies in health, which aim to confirm theoretical validity and internal consistency for use in real-world care practice and public health management contexts.
Research site
The study was conducted in three different UPA 24h, located in a specific capital city in the Northeast region of Brazil. This setting was chosen due to the significant local population, estimated at approximately 2.4 million inhabitants. The installed capacity of the network was also considered, which includes 11 level III UPA 24h, a number approximately three times higher than the coverage recommended for this level of care6.
The services included are representative within the context of the ECN, allowing the definition of parameters to evaluate the quality of care for the elderly in a real-world setting. To this end, a normative evaluation was carried out, in which the situation observed in the UPA 24h, considering dimensions of structure, process, and outcome, was compared to explicit standards defined by national guidelines and SUS regulations applicable to emergency services.
Based on this evaluation, judgments of conformity were derived by comparing current practices with the regulatory frameworks that guide the organization of the ECN. In this context, the normative evaluation was configured as a strategy to verify recognized quality standards20, based on indicators structured according to the Donabedian model, especially in recent research analyzing the quality of health services in Brazil7.
1st Stage: Literature review and selection of structuring references
To support the construction of the evaluative matrix, a literature review was conducted to ground the concepts of comprehensive healthcare for older adults, health evaluation, CV, and Donabedian’s Triad. Technical documents from the MH6, were consulted, in addition to discussions with managers and visits to the UPA 24h, to relate the documentary analysis to the Donabedian dimensions10.
The normative evaluation stage, guided by a structured script, examined, in an integrated way, the domains of structure, process and outcome within the UPA 24h context, which allowed identifying the priority needs of older adults within the scope of the ECN and subsequently to define the elements to be measured. On-site verification allowed assessment of the conformity of resources, care flows, and professional practices with current guidelines and, in an articulated manner, supported the selection of evaluative questions and indicators by dimension.
Concurrently, a scoping review was conducted between September and November 2022. The systematic search was carried out in the following databases: Virtual Health Library (VHL), Web of Science, PubMed, Scopus, CINAHL, and Cochrane. Inclusion criteria comprised full-text articles published between 2007 and 2022 in Portuguese, English, and Spanish that addressed the quality of care for older adults in urgent and emergency services. The final sample consisted of seven studies, whose most relevant findings indicated weaknesses in care for older adults in UPA 24h units, notably the lack of integration within the Health Care Network (HCN), high user demand, and the absence of risk management21.
Furthermore, issues related to the comprehensiveness of the HCN, structural inadequacies of the UPA 24h, user satisfaction with response time, and the need to sensitize the nursing team due to high demand, exposure to fall risks and opportunistic infections, and the lack of protocols and/or service evaluation instruments were identified21. The quality of care was addressed in an incipient manner in these studies, however, all findings were considered relevant to support the construction of the evaluative matrix, ensuring that the instrument addressed the critical areas identified in practice.
2nd Stage: Development of the evaluative indicator matrix
Based on the literature review, the conceptual properties of the construct were established, as well as the structuring of the dimensions and items of the evaluative matrix. The definition and organization of the items/indicators were based on guidelines and principles of the UPA 24h component, established in MH regulations (6).
Donabedian’s triad defined the construction and distribution of the matrix indicators10. Thus, the matrix was developed with 16 indicators, grouped into the following dimensions: Structure (7), Process (3) and Outcome (6). Furthermore, three attributes were considered for each indicator and dimension: (i) adequacy: to assess whether the indicator was appropriate for its objective and respective dimension; (ii) relevance: which assessed the relevance of the indicator for assessing the quality of care for older adults in UPA 24h; and (iii) clarity: to verify whether the concept and premises were understood and clearly expressed what was intended to be measured.
For methodological purposes, this research followed these sub-stages: (1) definition of focus; (2) selection of experts; (3) questionnaire structuring; (4) consultation rounds; (5) consolidation of results; and (6) presentation of the developed matrix16-17.
The study began with the definition of the construct and psychometric objectives of the instrument. Next, experts with experience and recognized expertise in the subject and in instrument validation were selected. The questionnaires were structured, reviewed, and adapted to the target audience. The two rounds allowed us to obtain the CV, as well as to collect suggestions for refinement, the results of which were consolidated by calculating the CVI and the AR. Finally, the methodological path, obtained indices, and the potential applications of the validated instrument were presented.
3rd Stage: Content validation of the matrix by a panel of judges
The selection of expert judges started with an online search on the Lattes Platform, in the “subject” tab, using the following terms: Public health service management; Emergency and Urgent Care Network; Older adults and quality of healthcare services. The search identified 10 professionals with profiles compatible with the inclusion criteria. Then, the snowball sampling technique was applied)22, which indicated another 20 experts, thus increasing the total number of eligible participants to 30. The formal invitation was sent to all selected expert judges, of whom 16 agreed to participate and composed the panel for the first round of content validation.
Inclusion criteria for the judges were: working in public health management, teaching and/or research at public universities in the fields of the Emergency Care Network, older adults, and health service quality, with a minimum of six months of experience in the role. Judges were excluded if they: (i) did not complete all stages of content validation; (ii) had conflicts of interest with the study; or (iii) failed to return the research instrument within the stipulated deadline.
The expert judges were contacted by email, and the Informed Consent Form (ICF) and the evaluation matrix, available on the Tyforms®, platform, were attached, with a 7-day deadline for return. Initially, 20 experts were defined, of whom 17 signed the ICF, confirming their participation. Finally, the panel comprised 16 experts who completed the content validation phase of the instrument. This number aligns with the scientific literature, which recommends a total of 10 to 30 specialists for panel composition16.
The instrument was presented to the experts in a semi-structured format, containing the title, instructions for completion, and a space for suggestions, criticisms, and comments. CV of the instrument was carried out in 2 phases16-17: the first assessed the dimensions of the matrix, and the second used a 5-point Likert-type ordinal scale 16-19, with the following levels: (5) strongly agree; (4) agree; (3) neither agree nor disagree; (2) disagree; and (1) strongly disagree. Concurrently, the items/indicators were evaluated according to the criteria of (i) appearance, (ii) applicability, (iii) clarity, (iv) readability, and (v) organization of the instrument.
Statistical evaluation was performed using the CVI, whose score was calculated by summing the “Strongly agree” and “Agree” responses of each expert and dividing by the total number of responses. The instrument’s validation was achieved through evidence extracted by the Delphi technique, which, after two rounds, obtained the consensus of the experts, ensuring methodological adequacy14-15. The agreement between the experts was assessed by the AR, with a cutoff point equal to or above 0.80, as recommended in recent studies16-18.
It should be noted that, for this study, no cutoff was adopted for the number of indicators to classify them as poor, regular, good, or excellent for defining overall quality. Instead, the 5-point Likert scale was applied, allowing analysis of responses according to the experts’ evaluation, with options 1 and 2 grouped as +1 and options 3 to 5 as -1. Using this recoding, agreement means ranging from -1 to +1 were calculated, identifying the degree of alignment among experts for each indicator. Therefore, the global evaluation did not depend on the simple sum of categories but on the statistical interpretation of the mean agreement scores assigned to the instrument.
To analyze qualitative aspects, expert comments were organized in an Excel® spreadsheet, grouped by dimension and their respective indicators, as well as by clarity, relevance, and adequacy. Data were cataloged as comments and contributions, defined as “applicable” or “not applicable,” with applicable contributions incorporated into the improvement of the matrix.
The parameters for each indicator were defined based on targets and standards established in Ministry of Health ordinances regulating the UPA 24h component within the ECN, as well as municipal management contracts establishing operational and care-related goals. This definition was supported by the bibliographic and regulatory review conducted in the first stage of the study, which gathered scientific evidence and official guidelines that informed the construction and parameterization of the evaluative matrix.
Ethical aspects
The study was submitted to the Research Ethics Committee of the School of Nursing at the Universidade Federal da Bahia and approved under opinion No. 6,271,357 on August 30, 2023. All procedures validated in the study comply with current ethical guidelines 23.
RESULTS
The CV stage of the research considered the CVI of the matrix, with the participation of 16 experts in the first round and nine in the second, among whom 4 were professors and 5 were managers. This allowed a thorough analysis of the indicators through conceptual and methodological validation of the instrument.
Regarding education and qualification of the sample, 50% held a master’s degree, 38% a doctorate, and 12% a specialization. The sample consisted predominantly of nurses (81%), with a 50% parity between professors and public managers from various levels of care, of whom 44% of whom were affiliated with Health Departments. The professional experience of the experts ranged from 5 to 25 years. Therefore, the matrix was submitted to the expert judges for evaluation, and its results were obtained by calculating the statistical scores of AR and CVI.
In the first round of the CV, the instrument achieved an overall AR of 0.97, with its dimensions as follows: Structure 0.98, Process 0.98, and Outcome 0.94. Regarding the aspects of adequacy, relevance, and clarity across the three dimensions of the instrument, an AR above 0.91 was observed in all dimensions, indicating consistent agreement among the experts.
It should be noted that the AR assessment was calculated based on (n=15) responses for most items, as one of the evaluators only met the adequacy criterion, which made it impossible to apply the complete methodology for this purpose. However, all aspects assessed in the first round presented a AR above 0.80.
As for the overall CVI, the matrix reached 0.95, reinforcing satisfactory convergence between the experts’ responses and the instrument’s content. The Structure, Process, and Outcome dimensions showed CVIs of 0.95, 0.98, and 0.93, respectively, suggesting alignment with the matrix’s initial design.
Considering the quantitative assessment, the matrix had all its indicators and dimensions validated (CVI ≥ 0.80) in the first round. However, the qualitative contributions of the experts motivated a second application of the instrument, which incorporated the suggested refinements.
After statistical analysis of the responses, the experts’ comments were examined qualitatively, totaling (n=145) records, of which (n=78) referred to the indicators of the Structure dimension; (n=26) to the Process dimension; and (n=41) to the Outcome dimension. However, most comments (n=106) focused on healthcare services, including criticisms and suggestions for improvement. Of the comments directed at the instrument itself, n=39 suggested ways to optimize it.
Following the first refinement, the applicability of relevant criticisms and suggestions was evaluated, considering data availability limitations or legal constraints. Ultimately, nine applicable contributions were identified, which were crucial for submitting the eight indicators most highlighted by the experts to a second round of CV.
In the second round, the number of participants decreased from n=15 to n=9, despite granting three extensions for response submission. This reduction is expected in studies involving instrument validation and does not compromise the quality of the expert panel or the execution of methodological steps.
In this round, the instrument presented a overall AR of 0.94, with dimensions distributed as follows: Structure 0.99, Process 0.96, and Outcome 0.88. The global CVI reached 0.93, with Structure at 0.95, Process at 0.96, and Outcome at 0.90. A comparative analysis of data from both rounds showed that the instrument maintained its statistical scores above the established cutoff across all three evaluative dimensions and overall, indicating that the new expert assessment validated the matrix.
However, an increase in AR values was observed for three indicators, confirming their relevance and the importance of retaining them in the instrument, according to the experts. The indicators with increased AR were: (i) Diagnostic resources (ECG, X-ray, and laboratory) - Structure, which increased from 0.93 to 1.0; (ii) Percentage of return visits to the emergency department within 72 hours for the same cause - Process, which rose from 0.93 to 1.0; and (iii) Percentage of stays over 24 hours in the UPA 24h - Outcome, which decreased slightly from 0.98 to 0.96.
Conversely, the Outcome dimension showed a slight decrease in the AR, from 0.94 to 0.88, attributed to the lower agreement obtained by two indicators: (i) Percentage of elderly eligible for the Home Care Service (HCS) under observation in the UPA 24h; and (ii) Percentage of user/elderly person satisfaction. Both decreased from 1.0 to 0.78, considering the AR of the Outcome dimension. Even so, this dimension maintained a AR of 0.88, remaining above the cutoff point of 0.80 established by the study.
Regarding CVI calculation in the second round, three indicators showed statistical improvement: (i) Percentage of older adults attended in UPA 24h - Structure, from 0.93 to 0.96; (ii) Percentage of return visits to the emergency department within 72 hours for the same cause - Process, from 0.93 to 1.0; and (iii) Percentage of stays over 24 hours in the UPA 24h - Outcome, from 0.88 to 0.96.
To evaluate the quality of care provided to older adults in the UPA 24h, the indicators in the matrix were classified into four categories: poor, fair, good, and excellent, according to targets defined by experts. For global interpretation, a synthesis criterion was adopted based on the predominance of these categories: quality is considered satisfactory when most indicators fall into “good” and “excellent,” fair when “fair” predominates, and unsatisfactory when a greater proportion falls into “poor”.
Given this context and considering that the instrument was approved in the first round while the second round focused on refining terminologies and parameters, as well as characterizing the indicators and their assumptions, the final version of the instrument was chosen, with the best CVI among the two rounds (Chart 1).
The validation of the matrix demonstrated methodological consistency, evidenced by the AR and CVI scores above the cutoff point of 0.80 in both rounds, confirming the conceptual coherence and stability of the instrument. The diverse qualifications of the expert judges, combined with the quantitative analysis and the careful examination of 145 qualitative comments, allowed for the incorporation of pertinent adjustments without altering the central structure of the matrix. Furthermore, the positive evolution of key items and the predominance of classifications in the “good” and “excellent” categories consolidated the overall evaluation as satisfactory. Thus, the aforementioned version is methodologically adequate for evaluating the quality of care provided for older adults in UPA 24h units.
DISCUSSION
The CV of the matrix showed that the proposed indicators reflect, in a representative way, fundamental dimensions of the quality of care for older adults in UPA 24h. The diversity of backgrounds and experience of the expert panel contributed to a comprehensive analysis of the items, an aspect highlighted by the literature as a pillar in the development of robust and contextualized assessment instruments24-25. The predominance of professionals with direct experience in services reinforces the practical adherence of the matrix to the care routines and organizational challenges of the ECN.
The structure of the matrix, organized according to Donabedian’s triad, made it possible to integrate structural elements, care practices, and clinical outcomes related to emergency care for the older population. This framework has been widely used to guide health evaluation processes, particularly in contexts of high organizational complexity, as it facilitates a sequential understanding of service conditions, care processes, and achieved outcomes10. The good acceptance of the indicators by the experts suggests adherence of this matrix to the evaluative tradition consolidated in the field of healthcare quality.
The adequacy of writing and language emerged as a central aspect in indicator validation, since clarity and objectivity are essential to avoid imprecision and to ensure coherence and relevance of the proposed instrument26. There was also geographical diversity with professionals working abroad (Spain and Canada) and in different Brazilian states (São Paulo and Bahia), which provided greater representativeness and broadened the variety of experiences in the CV24. Therefore, the combination of linguistic clarity and geographical diversity of the judges strengthened the CV, while incorporating multiple professional and contextual perspectives into the instrument.
The structure indicators encompass key elements for the organization and efficiency of UPA 24h units, such as the availability of physical resources, primary healthcare coverage, and staffing levels. These aspects have been associated with timely access, response time, and care resolvability in emergency contexts7,8. Incorporating these elements into the matrix makes it possible to capture structural conditions that directly influence the older adult’s experience within the unit and that require coordinated interventions among municipal management, regulation, and primary healthcare. Studies indicate that weaknesses in the integration between primary healthcare and UPA 24h units lead to overcrowding, increased length of stay, and recurrent visits7-9, reinforcing the importance of continuous monitoring of these indicators.
In the Process dimension, the need for strategies to prevent clinical deterioration and return visits to UPA 24h units became evident, such as monitoring protocols, response time, and recurrence of care for the same cause. Studies indicate that the adoption of tailored protocols and the implementation of geriatric training programs are key pillars in improving the quality of emergency care for older adults27. Thus, the sum of these strategies can minimize recurrences, prevent clinical complications, and improve care for older adults.
The Process dimension encompasses elements related to the standardization of care, the use of protocols, and the timeliness of decision-making. The literature highlights that the consistent adoption of clinical protocols and training in geriatric emergency care can reduce complications, improve risk triage, and decrease early return visits to services28-29. Thus, although fewer in number, the Process indicators capture aspects that directly influence safety and continuity of care, providing concrete parameters for reviewing care flows and implementing continuing education actions.
The Outcome indicators address clinical and experiential outcomes, such as discharges, transfers, deaths, 72-hour return visits, and older adult satisfaction. Recent studies recommend that quality assessments in geriatric emergency care include indicators that reflect both care outcomes and user perception, as these broaden the understanding of service quality30-31. The presence of an indicator aimed at identifying eligibility for the Home Care Service highlights the need for integration of the UPA 24h with other points of care, to guarantee continuity of care, reduce avoidable hospitalizations and avoid early readmissions.
The overall evaluation of the instrument was obtained through analysis of the CVI and AR across two rounds. In the first round, the CVI was 0.94, with values of 0.95 for the Structure dimension, 0.96 for Process, and 0.94 for Outcome, indicating consistency in agreement and in the content validation process. In a similar study, a technology developed in the field of nursing achieved a CVI of 0.9432, which reinforces the excellence of the indices achieved in this study.
The second round of CV had a reduced number of participants, which highlights the use of the Delphi technique as an ally in illustrating the potential of the panel composition14-15. In a similar study, there was a progressive reduction in participation between Delphi rounds, with (n=91) experts in the initial stage, (n=69) in the second and (n=56) in the third33. Despite the reduction in participants, the necessary representativeness was preserved to support the intellectual robustness of the panel, as well as the consolidation of consensus.
Of the sixteen assessed aspects, (n=9) presented an AR of 1.0, six of 0.93 and one of 0.85; that is, all experts considered the indicators valid to compose the instrument. A similar study found that instruments were validated with a AR above 0.90(34). In summary, the CV confirmed the adequacy of most indicators, which allows us to highlight the rigor in complying with the methodological recommendations and the consistency of the validated matrix.
The CV of the instrument was achieved in the first round. However, a second round was justified to incorporate the improvements suggested by the judges. In a recent study, the first round showed partial or total agreement above 0.70 for all items, eliminating the need for a second round35. Thus, the initial agreement, associated with the adjustments indicated by the judges, reflects the criticality of the analysis and confirms that AR values above 0.70 indicate methodological consistency of the evaluative matrix.
Regarding qualitative analysis, of the n=145 comments provided by the evaluators, 26% referred to improvements to the instrument within the Process dimension. Synthesizing this information was challenging, as it required discernment and experience from the researchers to select relevant contributions without unnecessarily increasing the size of the instrument15-16. Given this complexity, theoretically grounded analytical techniques were applied to qualify the information and improve the study.
It is noteworthy that the judges’ contributions were instrumental in ensuring the appropriate allocation of indicators across Structure, Process, and Outcome. Their input enabled textual adjustments and the reallocation of the indicator “Percentage of return to emergency within 72 hours for the same cause” to the Process dimension and the indicator “Percentage of stay longer than 24 hours in the UPA 24h” to the Outcome dimension. This reorganization preserved Donabedian’s theoretical framework10 and avoided misinterpretations, as supported by studies that classify outcome indicators as results36.
The predominance of Structure and Outcome indicators in the matrix highlights the centrality of adequate organizational resources and measurable outcomes to qualify care for older adults in UPAs 24h. The literature shows that qualifying care for older adults in geriatric emergencies requires an adapted physical environment, structured care protocols, and the need for ongoing education of the staff31. Another study emphasizes that defining indicators with measurement and support from psychometric properties makes it possible to identify critical weaknesses in care, thereby guiding planning and management domains in implementing strategies aligned with the real needs of older adults37.
The practical implications of the findings are significant. The matrix can be incorporated into the routine performance monitoring of UPA 24h units, supporting managers in identifying bottlenecks, planning improvements, and negotiating care-related targets. Its structure facilitates use in supervision processes, audits, continuing education, and comparative evaluation across units. For care teams, it serves as a tool to support reflection on care flows, work organization, and the qualification of care directed toward older adults.
Among the limitations, it is noteworthy that this research only evaluated the content validity of the indicators, without including empirical testing, reliability analyses, or construct validity. Future studies should apply the matrix in different contexts of the ECN, test its sensitivity, and evaluate the stability of the indicators over time.
Despite these limitations, the study contributes by proposing a theoretically anchored evaluative technology, grounded on scientific evidence and validated by experts. Thus, the matrix has the potential to support managers and professionals in defining more precise strategies oriented toward the needs of the older population, strengthening the quality and comprehensiveness of care within the UPA 24h component of the ECN.
CONCLUSION
The study developed an indicator matrix to evaluate the quality of care provided to older adults in UPA 24h units. The instrument presented satisfactory content validity, with statistical scores above the established cutoff point, indicating methodological consistency and theoretical alignment with the Donabedian Triad.
The constructed and validated matrix comprises 16 indicators distributed across the dimensions of Structure (7), Process (3), and Outcome (6), objectively covering installed capacity, care organization, and the effects produced in the care for older adults in UPAs 24h. Structure indicators involved care coverage, availability of professionals, ACCR rooms, and diagnostic resources; the Process indicators reflected the use of standardized protocols, decision-making within 6 hours, and 72-hour return visits; and the Outcome indicators included eligibility for HCS, discharges, transfers, prolonged stays, deaths, and user satisfaction. The consolidation of these components strengthens the matrix’s applicability by offering objective parameters for the continuous monitoring of care for older adults.
The matrix comprehensively integrated the dimensions of Structure, Process, and Outcome, enabling the identification of critical aspects of care and, consequently, guide interventions aimed at improving care quality. The developed instrument also represents an advance in quality management in urgent and emergency services, particularly within the UPA 24h component, by providing measurable parameters to support analysis and decision-making.
Finally, the application of the matrix in different ECN contexts and at other levels of care is recommended in order to test its applicability and expand the external validity of the findings. Future studies may explore indicator performance in time series and integrate this evaluative technology into health information systems, strengthening the monitoring of care quality.
REFERENCES
-
1. Nascimento JS, Santos DPS. Separação obrigatória dos nubentes maiores de 70 anos e sua (in)compatibilidade com as mudanças sociodemográficas. Rev Ibero-Am Humanid Cienc Educ. 2023;9(11):638. https://doi.org/10.51891/rease.v9i11.12281
» https://doi.org/10.51891/rease.v9i11.12281 -
2. World Health Organization (WHO). Decade of Healthy Ageing: baseline report [Internet]. Geneva: WHO; 2021[cited 2024 Oct 12]. Available from: https://www.who.int/publications/i/item/9789240017900
» https://www.who.int/publications/i/item/9789240017900 -
3. Instituto Brasileiro de Geografia e Estatística(IBGE). Projeções da População: Brasil e Unidades da Federação - Revisão 2024[Internet]. Rio de Janeiro: IBGE; 2024[cited 2024 Oct 12]. Available from: https://nada.ibge.gov.br/en/agencia-news/2184-news-agency/news/41065-populacao-do-pais-vai-parar-de-crescer-em-2042
» https://nada.ibge.gov.br/en/agencia-news/2184-news-agency/news/41065-populacao-do-pais-vai-parar-de-crescer-em-2042 -
4. Gómez-Galán R, López-Gil JF, Mendoza-Muñoz M, Carlos-Vivas J, Carvajal-Gil J, Muñoz-Bermejo L. Profile of users and adequacy of hospital emergency services in response to healthcare demand among population aged 65 years and over. Diseases. 2025;13(7):190. https://doi.org/10.3390/diseases13070190
» https://doi.org/10.3390/diseases13070190 -
5. Lamonato LCXL, Sarti TD, Almeida APSC. Effect of primary health care on the association between multimorbidity and emergency service utilization: National Health Survey, 2019. Rev Bras Epidemiol. 2024;27:e240062. https://doi.org/10.1590/1980-549720240062
» https://doi.org/10.1590/1980-549720240062 -
6. Ministério da Saúde (BR). Portaria GM/MS nº 1.997, de 24 de novembro de 2023. Altera as Portarias de Consolidação GM/MS nº 3 e nº 6, de 28 de setembro de 2017, para regular habilitação, homologação e financiamento dos serviços da Rede de Atenção às Urgências e Emergências no âmbito do SUS[Internet]. Brasília: Ministério da Saúde; 2023[cited 2024 Oct 12]. Available from: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2023/prt1997_27_11_2023.html
» https://bvsms.saude.gov.br/bvs/saudelegis/gm/2023/prt1997_27_11_2023.html -
7. Doll SCQ, Macieira C, Matta-Machado ATG, Borde EMS, Santos AF. Qualidade dos componentes pré-hospitalares fixos da Rede de Urgência e Emergência no Brasil: um estudo a partir de dados do PMAQ-AB e PNASS. Cad Saude Publica. 2022;38(8):e00009922. https://doi.org/10.1590/0102-311XPT009922
» https://doi.org/10.1590/0102-311XPT009922 -
8. Tofani LFN, Furtado LAC, Andreazza R, Bigal AL, Feliciano DGCF, Silva GRD, et al. A Rede de Atenção às Urgências e Emergências no Brasil: revisão integrativa da literatura. Saude Soc. 2023;32(1):e220122pt. https://doi.org/10.1590/S0104-12902023220122pt
» https://doi.org/10.1590/S0104-12902023220122pt -
9. Gunaga S, Carpenter C, Kennedy M, Hogan T, Shah MN, Carpenter CR, et al. A model for developing subspecialty clinical practice guidelines: the Geriatric Emergency Department Guidelines 2.0. J Am Coll Emerg Physicians Open. 2025;6(6):100247. https://doi.org/10.1016/j.acepjo.2025.100247
» https://doi.org/10.1016/j.acepjo.2025.100247 -
10. Donabedian A. The quality of medical care: how can it be assessed? JAMA. 1988;260(12):1743-8. https://doi.org/10.1001/jama.1988.03410120089033
» https://doi.org/10.1001/jama.1988.03410120089033 -
11. Correa JPC, Telles RCS, Guimarães MHD, Gomes DM, Fonseca ICM, Ferreira WLS, et al. Indicadores de Qualidade no Sistema Único de Saúde: abordagens para Avaliação da Eficiência e Eficácia dos serviços prestados. Interf J [Internet]. 2025 [cited 2025 Dec 12];11(2):2130-4. Available from: https://interferencejournal.emnuvens.com.br/revista/article/view/202
» https://interferencejournal.emnuvens.com.br/revista/article/view/202 -
12. Oliveira CCM, Silva PB, Andrade LM, Souza FL, Pereira RM, Santos JF, et al. Evaluation of mobile emergency service with mixed methods in Brazil. Cad Saude Publica . 2022;38(5):e00096221. https://doi.org/10.1590/0102-311XEN096221
» https://doi.org/10.1590/0102-311XEN096221 -
13. Caldas ACSG, Araújo RS, Medeiros PJ, Freitas MR, Aledo VS, Gama ZAS. Validação de indicadores de qualidade do cuidado cirúrgico no Sistema Único de Saúde. Rev Saude Publica [Internet]. 2023[cited 2025 Dec 12];57(1):27. Available from: https://www.scielo.br/j/rsp/a/zfZ8nDHvRrqV9z3K668TGFL/?lang=pt
» https://www.scielo.br/j/rsp/a/zfZ8nDHvRrqV9z3K668TGFL/?lang=pt -
14. Shang Z. Use of Delphi in health sciences research: a narrative review. Medicine. 2023;102(7):e32829. https://doi.org/10.1097/MD.0000000000032829
» https://doi.org/10.1097/MD.0000000000032829 -
15. Golz C, Hahn S, Zwakhalen SMG. Content validation of a questionnaire to measure digital competence of nurses in clinical practice. Comput Inform Nurs. 2023;41(12):949-56. https://doi.org/10.1097/cin.0000000000001037
» https://doi.org/10.1097/cin.0000000000001037 -
16. Silva BA, Silber PR, Brito WRS, Coimbra MA, Batista JF Calazans, et al. Processos de validação de instrumentos para área da saúde. Rev Eletron Acervo Saude. 2024;24(2):e14695. https://doi.org/10.25248/REAS.e14695.2024
» https://doi.org/10.25248/REAS.e14695.2024 -
17. Matos EP, Almeida DB, Freitas KS, Silva SSB. Construction and validation of indicators for patient safety in intrahospital transport. Rev Gaúcha Enferm. 2021;42:e20200442. https://doi.org/10.1590/1983-1447.2021.20200442
» https://doi.org/10.1590/1983-1447.2021.20200442 -
18. Almeida DB, Lima JMM, Silva GTR, Santos JLG, Erdmann AL, Almeida IFB. Validação de matriz avaliativa para avaliação dos planos de contingência de hospitais universitários brasileiros. Cienc Cuid Saude [Internet]. 2024[cited 2024 Oct 12];23:e7152. Available from: https://periodicos.uem.br/ojs/index.php/CiencCuidSaude/article/view/71527
» https://periodicos.uem.br/ojs/index.php/CiencCuidSaude/article/view/71527 -
19. Matos FR, Rossini JC, Lopes RFF, Amaral JDHH. Tradução, adaptação e evidências de validade de conteúdo do Schema Mode Inventory. Psicol Teor Prát [Internet]. 2020;22(2):18-38 [cited 2024 Oct 12]. Available from: https://pepsic.bvsalud.org/pdf/ptp/v22n2/pt_v22n2a02.pdf
» https://pepsic.bvsalud.org/pdf/ptp/v22n2/pt_v22n2a02.pdf -
20. Luz LA, Aquino R, Medina MG. Avaliação da qualidade da Atenção Pré-Natal no Brasil. Saúde Debate [Internet]. 2023 [cited 2024 Oct 12];42(esp 2):111-26. Available from: https://saudeemdebate.org.br/sed/article/view/835
» https://saudeemdebate.org.br/sed/article/view/835 -
21. Santos AA, Menezes TMO, Oliveira ES, Pessoa LSC, Moura HCGB, Pacheco FC. Atendimento pré-hospitalar oferecido ao idoso nos serviços médicos de emergência do Brasil: revisão de escopo. Rev Pró-UniverSUS. 2024;15(3);156-65. https://doi.org/10.21727/rpu.v15i3.4419
» https://doi.org/10.21727/rpu.v15i3.4419 -
22. Bockorni BRS, Gomes AF. A amostragem em snowball (bola de neve) em uma pesquisa qualitativa no campo da administração. Receu. 2021;22(1):105-117. https://doi.org/10.25110/receu.v22i1.8346
» https://doi.org/10.25110/receu.v22i1.8346 -
23. Conselho Nacional de Saúde (CNS). Resolução nº 738, de 1º de fevereiro de 2024. Dispõe sobre o uso de bancos de dados com finalidade de pesquisa científica envolvendo seres humanos [Internet]. Brasília: CNS; 2024[cited 2024 Oct 12]. Available from: https://bvsms.saude.gov.br/bvs/saudelegis/cns/2025/res_0738_22_01_2025.html
» https://bvsms.saude.gov.br/bvs/saudelegis/cns/2025/res_0738_22_01_2025.html -
24. Amaral JM, Almeida DB, Freitas GF, Tavares JPA. Validation of an evaluation matrix for the nursing process in the hospital context. Rev Gaúcha Enferm . 2024;45:e20230254. https://doi.org/10.1590/1983-1447.2024.20230254.en
» https://doi.org/10.1590/1983-1447.2024.20230254.en -
25. Gaia MA, Botelho EP, Ferreira GROB, Aben-Athar CYU, Parente AT, Sousa FJD, et al. LEARNHIV: development and validation of a mobile application for primary health care nurses focused on HIV care/prevention. BMC Nurs. 2023;22(1):419. https://doi.org/10.1186/s12912-023-01579-0
» https://doi.org/10.1186/s12912-023-01579-0 -
26. Duarte FH, Araújo NM, Silva SO, Leal NT, Costa TM, Alencar IG, et al. Validação do conteúdo de um recurso audiovisual para pessoas vivendo com HIV. Acta Paul Enferm. 2024;37:eAPE01361. https://doi.org/10.37689/acta-ape/2024AO0001361
» https://doi.org/10.37689/acta-ape/2024AO0001361 -
27. Pinardi E, Ornago AM, Bianchetti A, Morandi A, Mantovani S, Marengoni A, et al. Optimizing older patient care in emergency departments: a comprehensive survey of current practices and challenges in Northern Italy. BMC Emerg Med. 2024;24(1):86. https://doi.org/10.1186/s12873-024-01004-y
» https://doi.org/10.1186/s12873-024-01004-y -
28. Panitz LM, Freitas L, Andrade CLT, Gomes R, Pereira A, Silva J, et al. PROADESS: projeto avaliação do desempenho do sistema saúde. Cad Saude Publica . 2024;40(9):e00156023. https://doi.org/10.1590/0102-311XPT156023
» https://doi.org/10.1590/0102-311XPT156023 -
29. Brondani AC, Farias BG, Palombini AL. Indicadores qualitativos de processo e resultado no acompanhamento terapêutico. Rev Polis Psique. 2023;13(1):121-42. https://doi.org/10.22456/2238-152X.111288
» https://doi.org/10.22456/2238-152X.111288 -
30. Testa L, Richardson L, Cheek C, Morgan J, Patel R, Hughes T, et al. Estratégias para melhorar o atendimento a idosos que comparecem ao pronto-socorro: uma revisão sistemática. BMC Health Serv Res. 2024;24:178. https://doi.org/10.1186/s12913-024-10576-1
» https://doi.org/10.1186/s12913-024-10576-1 -
31. Endalamaw A, Khatri RB, Erku D, Nigatu F, Zewdie A, Wolka E, et al. Successes and challenges towards improving quality of primary health care services: a scoping review. BMC Health Serv Res . 2023;23(1):893. https://doi.org/10.1186/s12913-023-09917-3
» https://doi.org/10.1186/s12913-023-09917-3 -
32. Pizzolato AC, Sarquis LMM, Danski MTR, Cubas MR, et al. Validação de instrumento para Registro do Processo de Enfermagem no atendimento pré-hospitalar móvel de urgência. Rev Enferm UFSM. 2023;13:e11. https://doi.org/10.5902/2179769271997
» https://doi.org/10.5902/2179769271997 -
33. Niederberger M, Schifano J, Deckert S, Hirt J, Homberg A, Köberich S, et al. Delphi studies in social and health sciences: recommendations for an interdisciplinary standardized reporting (DELPHISTAR) - results of a Delphi study. PLoS One. 2024;19(8):e0304651. https://doi.org/10.1371/journal.pone.0304651
» https://doi.org/10.1371/journal.pone.0304651 -
34. Galli ACA, Silva MPC, Ued FV, Rezende LV, Ruiz MT, Contim D. Bundle para regulação da temperatura corporal de recém-nascidos maiores de 34 semanas: construção e validação. Rev Eletrôn Enferm. 2024;23. https://doi.org/10.5216/ree.v26.76948
» https://doi.org/10.5216/ree.v26.76948 -
35. Madeiro JR, Silva JLD, Sales ACVD, Souza EDS. Validação de conteúdo para um instrumento para avaliação de estudantes de Medicina em sessões tutoriais. Rev Bras Educ Med. 2021;45(03):e130. https://doi.org/10.1590/1981-5271v45.3-20200017
» https://doi.org/10.1590/1981-5271v45.3-20200017 -
36. De Rosis S. Performance measurement and user-centeredness in the healthcare sector: opening the black box adapting the framework of Donabedian. Int J Health Plann Manage. 2024;39(4):1172-1182. https://doi.org/10.1002/hpm.3732
» https://doi.org/10.1002/hpm.3732 -
37. Karimi-Dehkordi M, Eslami S, Mahdizadeh M, Heydari H, Kazemi S, Nouri Z, et al. Mapping quality indicators to assess older adult health and care in community, continuing, and acute-care settings: a systematic review of reviews and guidelines. Healthcare. 2024;12(14):1397. https://doi.org/10.3390/healthcare12141397
» https://doi.org/10.3390/healthcare12141397
Access to the dataset is available upon request from the corresponding author.
