Validation of instrument to assess software of patients ’ risk classifi cation

Objective: Validate the content of an instrument that assesses the quality of a software applied to the risk classifi cation of patients. Method: Methodological study, conducted in three stages: adaptation of the instrument, content validation through Delphi technique and pre-test. The results were analyzed through Content Validity Index, Overall Content Validity Index and Inter-rater Reliability. Results: The fi nal version of the instrument comprises 8 characteristics and 28 sub-characteristics, being 37 general questions to computer experts and nurse and 7 specifi c questions to computer experts, including 1 question and excluding 3 questions of the original instrument. We obtained Overall Content Validity of 92% and Inter-rater Reliability of 100% in the second Delphi round. Final considerations: The instrument has content validity, allowing to assess the technical quality and functional performance of the software applied to the risk classifi cation of patients. Descriptors: Classifi cation; Emergency Medical Services; Validation Studies; Validation of Computer Programs; Healthcare Quality. RESUMO Objetivo: Validar o conteúdo de instrumento que avalia a qualidade de um software aplicado à classifi cação de risco de pacientes. Método: Estudo metodológico, realizado em três fases: adaptação do instrumento, validação de conteúdo por meio da técnica Delphi e pré-teste. Os resultados foram analisados por meio do Índice de Validade de Conteúdo, Índice de Validade de Conteúdo global e Índice de Concordância Interavaliadores. Resultados: A versão fi nal do instrumento contempla oito características, 28 subcaracterísticas, sendo 37 questões gerais aos especialistas em informática e enfermeiro e sete questões específi cas aos especialistas em informática, com inclusão de uma questão e exclusão de três questões do instrumento original. Obteve-se percentual de Validade de Conteúdo Global de 92% e Índice de Concordância Interavaliadores de 100% na segunda rodada Delphi. Considerações fi nais: O instrumento possui validade de conteúdo permitindo avaliar a qualidade técnica e desempenho funcional de software aplicado à classifi cação de risco de pacientes. Descritores: Classifi cação; Serviços Médicos de Emergências; Estudos de Validação; Validação de Programas de Computador; Qualidade da Assistência à Saúde. RESUMEN Objetivo: Validar el contenido de instrumento que evalúa la cualidad de un programa (software) aplicado a la clasifi cación de riesgo de pacientes. Método: Estudio metodológico, realizado en tres etapas: la adaptación del instrumento, la validación de contenido por medio de la técnica Delphi y la prueba previa. Los resultados fueron analizados por medio del Índice de Validez de Contenido, Índice de Validez de Contenido Global e Índice de Concordancia Interevaluadores. Resultados: La versión fi nal del instrumento contempla ocho características, 28 subcaracteristicas, siendo 37 cuestiones generales a los expertos en informática y enfermero y siete cuestiones específi cas a los expertos en informática, con inclusión de una cuestión y exclusión de tres cuestiones del instrumento original. Se obtuvo porcentual de Validez de Contenido Global del 92% e Índice de Concordancia Interevaluadores del 100% en la segunda ronda Validation of instrument to assess software of patients’ risk classifi cation Validação de instrumento para avaliar software de classifi cação de risco de pacientes Validación de instrumento para evaluar el programa (software) de clasifi cación de riesgo de pacientes

Processes (PROCenf-USP) through an instrument that contemplated the last resolution specific to this type of assessment (3) .

OBJECTIVE
To validate the content of an instrument that assesses the quality of a software applied to the risk classification of patients.

METHOD Ethical aspects
This study was approved in April 27, 2016 by the Research Ethics Committee of the Londrina State University (UEL).

Study design, location, and period
This is a methodological and applied study, with a quantitative approach, carried out from April to October 2016, consolidated in three stages, namely: instrument adaptation, validation of content and pre-test.The study was held in a medium-sized public secondary hospital, located in the State of Paraná, Brazil.

Study protocol
For validation of this instrument we opted for the Delphi technique, a systematized method of judgment of information intended to reach the consensus of opinions about a particular subject, regarding the knowledge of a committee of experts, through validations articulated in rounds, maintaining anonymity. (15)Content validation is held by questioning various experts who analyze the representativeness of the items that integrate the object of analysis (16) .
The instrument chosen in its original version aims to assess the software quality of the Electronic Document System of Nursing Processes PROCenf-USP ® (3) , which includes the latest update from the International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC), specific to the assessment of the software.

INTRODUCTION
The quality of health information is a global concern.Countries such as the United States of America, Canada and England promote initiatives for the safety in the conception, acquisition and deployment of health information technology (1) .Advances in access to information, increased accuracy of documentation, deployment of evidence-based practice, cost reduction with return on investment, as well as improvements in quality of care and satisfaction of employees, are evidence of international experiences in the use of computerization in health (2) .
In Brazil, the use of computer systems to management has increased exponentially in different areas.In the work process of nursing, computer science improves more and more through the development and evaluation of tools, processes and structures that assist nurses in care management (3)(4) .
In emergency hospital services, computerization has been used for the Assessment with Risk Classification of patients.International literature brings two Assessment and Risk Classification protocols in electronic format: the Manchester Triage System and the Andorrá Triage Model (5) .In Brazil, Manchester is the most known protocol, which has inspired the computerization of the National Protocol of Risk Assessment and Classification, implemented by the Ministry of Health through the ordinance 2048/2002 (6) .However, no scientific evidences were found on the evaluation of technical quality and functional performance of software used for this computerization, both of national and international protocol.
The assessment of a software is critical to identify the weaknesses and limitations of the product, analyze its performance and diagnose the need for adjustments (7) .This assessment is consolidated by specific regulations, such as the International Organization for Standardization (ISO), International Electrotechnical Commission (IEC) and Brazilian Association of Technical Standards (ABNT), which proposed two Brazilian Standards (NBR), ISO/IEC 14598 and ISO/IEC 9126, regarding the quality of software products (8)(9) .In 2011, updates of these regulations gave rise to the standards ISO/IEC 25010 -System and Software engineering -System and software quality models; and ISO/IEC 25040 -System and Software engineering -Evaluation process (10)(11) .Such standards have been used to create software and its assessment in different contexts of Brazilian nursing (3,7,(12)(13)(14) .
In this sense, we identified on our work practice the need to assess the technical quality and functional performance of a software used to perform the Risk Assessment and Classification of patients in a Hospital Emergency Service (HES).To do so, our study was based on a recent survey that applied such methodology in an Electronic Document System of Nursing Validation of instrument to assess software of patients' risk classification Pegoraro LGO, Gvozd R, Haddad MCFL, Vannuchi MTO, Silva LGC, Rossaneis MA.
The author of this instrument granted permission for its adaptation and validation to assess the software of Risk Classification of patients.
The first stage of the study was the adaptation of the original instrument to the context of Assessment with Risk Classification of patients.In the second stage, we proceed to content validation of the instrument adapted through Delphi technique.Two Delphi rounds were needed.In the first, the committee was composed of 6 experts; of these, 3 computer experts with experience in creating software and computerization in health area and 3 nurses, 2 doctors and 1 master, both with experience in validation and adaptation of instruments.The same experts were invited for the second round, but only 5 responded to the request.
After prior contact with the experts by e-mail, the following documents were sent: an Informed Consent Form in electronic format, a summarized protocol for the presentation of the research project, the instrument adapted and instructions for completion and assessment of the questions to be reviewed by them.It was stipulated a deadline of 15 days for the return of the analyzed material.
Experts assessed each item of the instrument in +1 (suitable), -1 (suitable with amendments) or 0 (unsuitable).At the end of the assessment, it was possible to calculate a Content Validity Index (CVI) and the Overall Content Validity Index, which measure the proportion or percentage of evaluators who are in agreement on certain aspects of the instrument and its items, being considered valid the ones that obtained minimum percentage of 80% of agreement between the experts (17)(18) .
The Reliability Index or Inter-rater Reliability, which is intended to assess the extent to which the evaluators are reliable when assessing the items outside the context studied, was calculated for each dimension of the instrument, by dividing the number of items that scored above 80% of agreement between evaluators by the total number of items of each dimension (17)(18) .
After content validation, the third stage of the study was held by pre-testing the adapted instrument.The selection of individuals for this stage occurred through purposive sampling at the institution studied, being composed by 1 computer expert, 3 nurses with experience in computerized Assessment of Classification Risk of patients and 3 nurses without experience in this activity.In this stage, participants were asked about problems regarding the understanding of each item and terms difficult to be understood; they assessed the questions through questionnaire of Practicality of Instrument (19) .

Analysis of results and statistics
Data were tabulated in the software Microsoft Excel 2010 and analyzed from reflective reading and descriptive statistics.

RESULTS
In the process of adaptation of the instrument, researchers chose to combine some questions that complemented each other to better understand the content.Sub-characteristics "Adaptability" and "Installability" were grouped into a single question."User Interface Aesthetics," "Time," and "Integrity" had questions grouped each one within its own sub-characteristic.
The content validation procedure consolidated the final version of the instrument with 8 characteristics, 28 sub-characteristics and 44 questions.The characteristics Functional Adequacy, Reliability, Usability, Performance Efficiency, Compatibility, and Safety appear in 37 questions and are aimed at computer experts and nurses.The characteristics Maintainability and Portability are included in 7 specific questions to computer experts.One question was included and 3 were excluded of the original instrument.From the first Delphi round, we identified the need to reformulate some questions of the instrument, justifying the second Delphi round.
Table 1 presents the Content Validity Index (CVI) of each characteristic and sub-characteristic of the instrument, the Inter-rater Reliability (IRR) and the Overall Content Validity Index, verified in the first and second rounds.According to the assessment of experts regarding the agreement and representativeness of the items in the instrument, the overall CVI in first Delphi round was 70%, reaching 93% in the second round.
The characteristics Usability and Portability obtained the best percentages, with CVI above 80% in the first Delphi round.Safety, Reliability, Compatibility and Maintainability presented the lowest percentages of CVI (53%, 62%, 63% and 63%, respectively).
Regarding reliability, just the Portability characteristic reached satisfactory index in the first Delphi round.All the others reached IRR of 100% in the second round.
Of the 30 sub-characteristics containing the instrument in the first round, 15 (50%) reached CVI above 80%, being 5 with 100% of compliance.Availability, Accountability, and Authentication obtained the lowest percentages of CVI (0%, 0% and 33%, respectively).In the second round, the Accountability and Authentication sub-characteristics were excluded from the instrument on the recommendation of the experts.
In Chart 1 are presented the suggestions of the committee of experts on the items considered "suitable with amendment" and "unsuitable," obtained in the first and second Delphi round.
After content validation, the instrument was submitted to a pre-test conducted with 7 participants: 1 computer expert, 3 nurses with experience in computerized Assessment of Classification Risk of patients and 3 nurses without experience in this activity.The results of this stage showed ease of understanding of the items of the instrument and pertinence of the proposal of software assessment; there are no suggestions for adjustments in the items regarding the instrument.Two participants suggested replacing the words Disagree and Agree by Yes and No expressions, but the researchers considered this change inappropriate.The time demanded by participants for completing the instrument was approximately 18 minutes to each.

DISCUSSION
The option of adapting and validating an instrument to assess the computerization of the Assessment with Risk Classification in Hospital Emergency Services (HES), using a tool that indicates the level of technical quality and functional performance, was based on the fact that, so far, no instrument was able to quantify the opinions and impressions of health professionals regarding technical quality and functional performance of this computerized instrument.
In this stage, we decided to group the questions of the subcharacteristics Adaptability and Installability, presenting them in a single question.The sub-characteristics "Time" and "Integrity" had their questions grouped, each one within its own sub-characteristics.Such changes do not generate questions from the experts, nor concerns regarding clarity and representativeness, reaching a CVI above 80% since the first Delphi round.
However, the sub-characteristic "Interface Aesthetics," which also had its questions grouped to better research, presented CVI of 67% in the first round.The experts assessed it as "suitable with amendments" and suggested replacing the term "graphic design" by "software design" and adding which characteristics must be assessed.Such suggestions were accepted, reaching a CVI of 80% in the second round.
In the adaptation of the instrument, through the analysis of the results and reflective reading of the suggestions and comments of the committee of experts, it was possible to verify problems regarding clarity and representativeness in the instrument.Considering the method used in this research, which required IRR of 100% and a minimum CVI of 80%, the results obtained in the first Delphi round showed the need for adjustments of the instrument.
In the first Delphi round, 50% of the sub-characteristics that compose the instrument did not achieve the minimum CVI recommended by the literature and adopted in this research (17)(18) .Adjustments were made on the questions of the items indicated by the experts, through reformulations of writing on the questions, such as inversions of words or replacement of items by synonyms, which allowed better understanding of the items (21) .After this stage, we proceed to the second Delphi round to ensure that the adjustments made would obtain the clarity and agreement required.
In the first Delphi round, an expert suggested adding on the "Recognition of Suitability" sub-characteristic, of the Usability characteristic, a question related to the occurrence of training for the use of the software.The suggestion of the experts was accepted, since it was considered relevant within the context of the use and assessment of technical quality and functional performance; the other questions of the sub-characteristic received suggestion of reformulations and standardization of writing, which were accepted and submitted to a second round of assessment by the committee, obtaining CVI compliance percentage > 80%.
In this context, we consider that the incorporation of new technologies in the work process requires skills that must be acquired by health professionals, to have success and efficiency in the use of these tools.In this sense, training courses are important strategies, a praxis for knowledge production and success in the appropriate use of technologies, with direct impact on the quality of care provided (22)(23)(24) .The failure of the training courses or their absence may influence on the ability and desire of professionals to engage in activities of development, applicability and transformation of practices.
Still on the content analysis of the instrument, the characteristics Functional Fitness, Maintainability, Recoverability, Learnability, Capacity and Testability achieved percentage ≥ 80% in the first Delphi round, but were respectively assessed as "unsuitable" by an expert and "suitable" by the others; although there is no suggestion of reformulation and reaching favorable percentage of CVI, they were forwarded to the second round and reached percentage of 100%.
However, the "Fault tolerance," "Availability," and "Coexistence" sub-characteristics did not reach percentage of 80% or more in the first Delphi round, and were respectively assessed as "unsuitable" by an expert and "suitable" by the others.Changes were made on the items indicated, considering their relevance.They were forwarded to the second Delphi round, reaching favorable percentage of CVI, mostly 100% of agreement; and considered as valid, trustworthy and representative content, as advocated by the reference adopted in this study (15)(16)(17)(18)(19) .
On the other hand, the "Accountability" and "Authentication" sub-characteristics were removed from the instrument in the first Delphi round, since both were assessed as "unsuitable" by six experts, being observed problems of redundancy that can influence the results, according to the committee, justified as contemplated in others sub-characteristics of the instrument.
In the end, the indexes from the content validation process indicated high reliability and trustworthiness of the instrument and made it possible to assess the technical quality and functional performance of software applied to computerized risk classification of patients in EHS, a tool that helps professionals in the health care and that can be applied to several processes of computerization in health.

Study limitations
We considered as limitations of this study the number of experts in computer science that would contemplate the inclusion criteria established, i.e., with experience in computerization in health, given the shortage of these professionals on the staff of most health services.Health Informatics in Brazil is an area in exponential growth that has attracted professionals from diverse academic segments in search of specializations that confer competence to act.However, we found a restriction of specialization courses in this area and, among these, there are differences in their course syllabus, corroborating the limitations of our study (25) .

Contributions to the Nursing field
The validated instrument brings contributes to the health sector in the acquisition of a tool that aims to assess the efficiency of software, qualify, promote improvements in its development and construction through health institutions, considering that this instrument can be continuously optimized and adapted to other computerized work processes.We emphasize the development of technical and scientific skills to healthcare professionals, specifically in the nursing field, with incentives to an entrepreneurial attitude, which is still found Validation of instrument to assess software of patients' risk classification Pegoraro LGO, Gvozd R, Haddad MCFL, Vannuchi MTO, Silva LGC, Rossaneis MA. empirically in many hospitals.These professionals are assisted by the exponential growth of technologies whose purpose is to meet their daily needs in different contexts of their practice, facilitating the work time and improving the care provided.

FINAL CONSIDERATIONS
The content validation procedure consolidated the final version of the instrument with 8 characteristics, 28 sub-characteristics and 44 questions.The Functional Adequacy, Reliability, Usability, Performance Efficiency, Compatibility, and Safety characteristics appear in 37 questions and are aimed at computer experts and nurses.The Maintainability and Portability characteristics are included in 7 specific questions to computer experts.
The overall CVI in the first Delphi round was 70%, reaching 93% in the second round.Of the 30 sub-characteristics containing the instrument in the first round, 15 (50%) reached CVI > 80%, being 5 with 100% of compliance.Availability, Accountability and Authentication obtained the lowest percentages of CVI (0%, 0% and 33%, respectively).In the second round, the "Accountability" and "Authentication" subcharacteristics were excluded from the instrument on the recommendation of the experts.
Regarding reliability, just the Portability characteristic reached the satisfactory index in the first Delphi round.All the others reached IRR of 100% the second round.
With two Delphi rounds, we obtained satisfactory percentages of Overall CVI and IRR, inclusion of 1 question and exclusion of 3 questions presented in the "Accountability" and "Authentication" sub-characteristics.In addition, in the pretest, the instrument was considered suitable regarding the clarity and objectivity in the items.
Due to high levels achieved in the content validation of the instrument, it is important that EHS which use the computerized Assessment with Risk Classification have this instrument as a tool for measuring, reliably, the technical quality of such software, to obtain, on its results, diagnoses that enhance this tool and even the construction of new software.We also consider that this instrument can be continuously optimized and adapted to other computerized work processes in health.
To do so, we suggest new research.on the theme, with expectation that the instrument be released and used for health institutions to qualify, enhance and promote improvements on the development and construction of software that facilitate the process of work in health area, the development of technical and scientific skills and the incentive of this entrepreneurial practice.

Table 1 -
Percentage of agreement between the experts in relation to the Content Validity Index (CVI) of each characteristic and sub-characteristic of the instrument, Inter-rate Reability (IRR) and Overall Content Validity Index in the first and second Delphi rounds, Londrina, Paraná, Brazil, 2016 Validation of instrument to assess software of patients' risk classification Pegoraro LGO, Gvozd R, Haddad MCFL, Vannuchi MTO, Silva LGC, Rossaneis MA.
Note: *AACR -Assessment with User Embracement and Risk Classification.