The development and evaluation of software to verify diagnostic accuracy

Abstracts

Este artigo descreve o desenvolvimento e avaliação de um software que verifica a acurácia diagnóstica de alunos de enfermagem. O software foi baseado num modelo que utiliza conceitos da lógica fuzzy, em PERL, banco de dados MySQL para acesso pela internet e a classificação NANDA-I 2007-2008. Avaliou-se a qualidade técnica e a usabilidade do software utilizando instrumentos específicos. A atividade proposta no software possui quatro etapas nas quais o aluno estabelece valores de relação entre diagnósticos de enfermagem, características definidoras/fatores de risco e casos clínicos. Os valores de relação determinados pelo aluno são comparados aos de especialistas, gerando escores de desempenho para o aluno. Na avaliação, o software atendeu satisfatoriamente as necessidades de qualidade técnica e, segundo os alunos, trouxe benefícios ao aprendizado, podendo transformar-se em uma ferramenta educacional no ensino do diagnóstico de enfermagem.

Diagnóstico de enfermagem; Tecnologia educacional; Lógica fuzzy; Informática em enfermagem

ORIGINAL ARTICLE

The development and evaluation of software to verify diagnostic accuracy

Rodrigo Jensen^I; Maria Helena Baena de Moraes Lopes^II; Paulo Sérgio Panse Silveira^III; Neli Regina Siqueira Ortega^IV

^IRN. Master's. Doctoral student, Nursing Graduate Program at the University of Campinas, Faculty of Medical Sciences. Campinas, SP, Brazil. [e-mail]

^IIRN. PhD. Associate Professor, University of Campinas, Faculty of Medical Sciences, Nursing Department. Campinas, SP, Brazil. [e-mail]

^IIIPhysician. PhD. Associate Professor, University of São Paulo, School of Medicine, Department of Pathology. São Paulo, SP, Brazil. [e-mail]

^IVPhysicist. PhD. Researcher at the University of São Paulo, School of Medicine, Department of Pathology. São Paulo, SP, Brazil. [e-mail]

ABSTRACT

This article describes the development and evaluation of software that verifies the accuracy of diagnoses made by nursing students. The software was based on a model that uses fuzzy logic concepts, including PERL, the MySQL database for Internet accessibility, and the NANDA-I 2007-2008 classification system. The software was evaluated in terms of its technical quality and usability through specific instruments. The activity proposed in the software involves four stages in which students establish the relationship values between nursing diagnoses, defining characteristics/risk factors and clinical cases. The relationship values determined by students are compared to those of specialists, generating performance scores for the students. In the evaluation, the software demonstrated satisfactory outcomes regarding the technical quality and, according to the students, helped in their learning and may become an educational tool to teach the process of nursing diagnosis.

Descriptors: Nursing diagnosis; Educational technology; Fuzzy logic; Nursing informatics

RESUMEN

Este artículo describe el desarrollo y evaluación de un software que verifica la exactitud diagnóstica de alumnos de enfermería. El software se basó en un modelo que utiliza conceptos de lógica fuzzy, en PERL, banco de datos MySQL para acceso por Internet y la clasificación NANDA-I 2007-2008. Se evaluó calidad técnica y usabilidad del software utilizando instrumentos específicos. La actividad propuesta en el software consiste en cuatro etapas, en las que el alumno establece valores de relación entre diagnósticos de enfermería, características de definición/factores de riesgo y casos clínicos. Los valores de relación determinados por el alumno son comparados con los de especialistas, generando puntajes de desempeño del alumno. En la evaluación, el software atendió satisfactoriamente las necesidades de calidad técnica y mostró que, en la percepción de los alumnos, trajo beneficios de aprendizaje, pudiendo transformarse en una herramienta educativa en la enseñanza del diagnóstico de enfermería.

Descriptores: Diagnóstico de enfermería; Tecnología educacional; Lógica difusa; Informática aplicada a la enfermería

INTRODUCTION

A Fuzzy Logic-Based Model to Assess Diagnostic Accuracy was developed⁽¹⁾. Through this model, degrees of relationship (degrees of membership) are established among defining characteristics (DC), risk factors (RF), nursing diagnoses (NDx) and degrees of presence of defining characteristics/risk factors in a given clinical case.

A software program was developed based on this model. It evaluates the accuracy of a nursing diagnosis and can be used in teaching. Through this software, one can compare the degrees of membership established by students with those established by experts. The software program seeks to encourage understanding and the practice of diagnostic rationale, in addition to offering the professor an objective method to evaluate the students' diagnostic accuracy.

LITERATURE REVIEW

A nursing diagnosis depends on the interpretation of human behavior related to health, which creates complexity with a great risk of low accuracy⁽²⁾. To improve accuracy, teaching strategies should be encouraged that focus on diagnostic rationales based on the correct identification of signs and symptoms and etiology of diagnoses⁽³⁾.

The use of fuzzy logic can be an alternative to the development of methods to show students how to direct their rationale to generate an accurate diagnosis. Fuzzy logic could be used to help experts consider how to establish their decisions and even how to assign weight to each one of the rules used in this process. This theory can help experts to verbalize their decision-making process and this understanding could be transmitted from professors to students⁽⁴⁾.

Fuzzy logic has been applied in the development of computer programs used for diagnoses that involve imprecision. Diverging from traditional logic, fuzzy logic permits a gradual transition (degree of membership) between the sets [0,1] where for each element xU, µA (x) indicates the degree to which x is a member of the set A. In the concept of partially true values, fuzzy logic theory is used where there is imprecision and uncertainty. The symbolic system allows for the use of linguistic terms based on uncertainty⁽⁵⁾.

This paper describes the development and evaluation of a software program that verifies the diagnostic accuracy of nursing students.

METHOD

Methodological study conducted in a public university in the state of São Paulo, Brazil, approved by the Research Ethics Committee at the institution (protocol No. 594/2008). All the participants signed free and informed consent forms before the study was initiated. The software was developed and evaluated between March 2009 and March 2010.

The concepts of fuzzy logic⁽⁵⁾, fuzzy maximum-minimum composition⁽⁶⁾, fuzzy aggregation operation⁽⁷⁾, and the Fuzzy Logic-Based Model to Assess Diagnostic Accuracy⁽¹⁾ were applied in the software development.

The programming language Practical Extraction and Report Language (PERL) and the MySQL database were used so that the program could be accessed through the World Wide Web (WWW). NANDA International Nursing Diagnoses (NANDA-I) version 2007-2008⁽⁸⁾ was also used.

The program was evaluated in terms of technical quality and usability, complying with Brazilian standard ISO/IEC 14598-6⁽⁹⁾, which recommends a minimum of eight evaluators. The instrument Sperandio⁽¹⁰⁾ was used to evaluate the program's technical quality, which complies with the Brazilian standard ISO/IEC 9126-1⁽¹¹⁾. This standard evaluates the software's functionality, reliability, usability, efficiency, maintainability and portability, while each characteristic is composed of sub-characteristics that total the items evaluated by the experts.

Eight experts with backgrounds in Analysis and Development of Systems or Computer Science performed the evaluation. The experts were provided with access registration to the software and general guidance on the evaluation process. The evaluation of experts was individually carried out and there was no interference from researchers. The experts received a manual with information on how the software was developed, detailed specification of each item under evaluation and instructions for the evaluation process.

Two studies^(12-13) were the basis for the questionnaire developed to evaluate the software's usability. The questionnaire was tested with six students who took the Nursing Diagnosis course in 2008. The instrument's comprehensiveness, clarity, difficulties and general appearance were evaluated. No changes were required.

Once tested, the questionnaire was applied to 27 nursing students who took the course in 2009 and used the software. The students used the software in October and November 2009 and the questionnaire was applied at the end of the course.

RESULTS

Description of the software

The software to evaluate diagnostic accuracy, called Fuzzy Kitten, is implemented and available in Portuguese and will be soon also available in English. This software includes secure access and requires the use of individual logins and passwords. The software can be accessed from any computer connected to the Web.

The presentation of the software to users follows a logical sequence and permits students to add registration information, communicate with their group or the system's administrator, recover attempts of activities to analyze them again, pause the activity and see performance scores (Figure 1).

When the activity is initiated, the student is presented with a page containing instructions. When each stage is concluded, the student cannot go back to previous stages until the activity is over. When the student performs an activity proposed by the software Fuzzy Kitten, s/he can see the linguistic variables to which the system assigns numerical values.

The time spent by the student to complete the activity is shown on the screen and s/he can choose to pause the activity. The professor can see the date, time and duration of each stage performed by students.

If access is interrupted, when the student restores access, s/he is directed to the interrupted stage, and the chronometer is activated at the time the interruption occurs to avoid loss of data, harm to students or the need to reinitiate the activity from the beginning.

All the activities performed remain available for students or professors to consult in the future. When the student repeats an activity, s/he can recover previous attempts and analyze them again for a new attempt. All the scores obtained are stored and remain available for the student to compare performance over all her/his attempts. The activity proposed to the student is composed of four stages described as follows.

First step

Before the student learns the clinical case to be analyzed, a list with some nursing diagnoses (NDx) is presented (some of them are related to the clinical case and others are not). When a NDx is selected, the student is directed to a table with some defining characteristics (DC) and risk factors (RF), which may be presented in the clinical case, where the degree of relationship among each DC/RF and the studied NDx has to be indicated.

Each linguistic variable corresponds to the following values of relationship (degree of membership): strongly related (SR)=1; related (RE)=0.75; moderately related (MO)=0.50; weakly related (WE)=0.25; not related (NR)=0. When the student does not define the relationship, the option not defined (ND) remains selected. The student establishes the relationships among all the nursing diagnoses and defining characteristics/risk factors that will be used in the activity. At the end of this step, a table with the intensity of relationships among the NDx and DC/RF is presented in a gradient blue, where the color's intensity represents the intensity of relationship. In this step, the general knowledge of the student concerning the manifestation of nursing diagnoses and NANDA-I taxonomy is evaluated.

Second step

A clinical case is presented at this point. The clinical case is presented to students at the second step of the activity so it does not influence the first step.

Third step

Students are asked to determine the patient's health condition based on the clinical case, that is, the signs and symptoms or risk factors the patient presents, identifying the DC/RF present in the case. The DC/RF presented at this point are the same as those presented in Step 1. Degree of presence (degree of membership) to be assigned: present (PR)=1; possibly present (PP)=0.75; I do not know (DK)=0.5; possibly absent (PA)=0.25; absent (AB)=0. At this step the student's ability to identify DC/RF presented in the clinical case is evaluated.

Fourth step

Students are asked to determine and classify the nursing diagnosis based on the patient's health condition. The NDx presented at this point are the same previously presented in the first step of the activity. The corresponding classification and score are: real/ wellness/ syndrome (A)=1; of risk (R)=1; possible (P)=0.5; undetermined (U)=0. The nursing diagnoses real, wellness, syndrome or of risk are assigned a value of 1 because they need the DC or RF to be in evidence. An additional blank space is added where another NDx can be included besides those listed. At this point, the student's ability to determine NDx and correctly classifying them is evaluated.

At the end of the activity, the software analyzes the values of relationships assigned by the student in the first and third steps of the activity. Using the Fuzzy Logic-Based Model to Assess Diagnostic Accuracy⁽¹⁾, which applies fuzzy maximum-minimum composition, the software presents new relationship values for the NDx group in addition to values determined by the student at the fourth step. The new NDx values result from relationships the student determines in the activity's first and third steps. For these NDx to be determined, an inference process similar to a matrix multiplication operation is performed, replacing the algebraic sum operator by a maximum operator while a minimum operator replaces the multiplication operator. Therefore, minimum values are selected among the degrees of membership of the activity's first step and the respective degree in the third step for each DC/RF. Then a maximum value is selected among all the minimum values previously determined for each diagnosis. At the end of this process, degrees of membership are defined for each NDx. The process is concluded when the final diagnoses are determined, such as those that reach a value of 1 (on a scale of 0 to 1) or the maximum value of the distribution of diagnostic possibilities. This analysis concludes the activity considering new values of relationship to the NDx group, that is, based on the student's rationale process throughout the activity.

The software has three clinical cases with NDx defined by experts in a previous study⁽¹⁴⁾. Participants of a research group on NDx performed the four steps of the activity and reached a consensus on the respective degrees of membership. Nurses with academic degrees participated in the group: Doctor and Researcher in NDx (1), doctoral students whose dissertation were related to NDx (3), Doctoral nursing student (1), Master's degree in education (1), Master's nursing student (1); experts (2), and undergraduate student (1). There were among the participants one researcher and five professors. The participants of the research group were the experts who established the relationships in the activity. The values assigned by the experts permit comparison with those established by the students when they performed the activity, so that performance scores can be generated for students. The performance score are presented in terms of a percentage (0 to 100%), indicating how closely the student's evaluation matched the experts' opinions.

Four performance scores are established:

Score A - Performance in establishing the degree of relationship between DC/RF and NDx;

Score B - Performance in identifying the degree of presence of DC/RF in the clinical case;

Score C - Performance in determining and classifying the NDx;

Score D - Performance in determining the NDx indicated by the Fuzzy Logic-Based Model to Assess Diagnostic Accuracy⁽¹⁾, which applies the fuzzy maximum-minimum composition.

The student's correct and incorrect answers obtained in each step of the activity are presented on the scores page. Comparison between scores C and D allows students to verify whether they only classified the NDx correctly or whether the trajectory of their rationale process related to knowledge of NANDA-I taxonomy, manifestation of NDx and identification of signs and symptoms in the clinical case, led them to the correct NDx (Figures 2 and 3).

Software evaluation

The software was evaluated in terms of technical quality and usability. Eight experts evaluated the software in terms of technical quality; the evaluation results are presented in Table 1. The experts evaluating the software's technical quality had an academic education in Computer Sciences (5/8) or Analysis and Development of Systems (3/8) and some were Master's (4/8) or doctoral (2/8) students.

The experts selected the option disagreement or it does not apply in various aspects of the technical quality evaluation explaining they either did not know the standards and laws that apply (item FUNCTIONALITY: conformity), given the absence of failures when using the software (items RELIABILITY: tolerance of failures and MANTAINABILITY: analyzability), for not having access to the source code (item MANTAINABILITY: stability) or for it being impossible to evaluate the item (item PORTABILITY: ability to replace).

The experts suggested some esthetic changes to the color gradient used in the screen of relationships between the NDx and DC/RF; with the possibility of modifying figures in the menu, adding and removing some of the menu buttons, and following a hierarchical model; organizing the layout of features; provided suggestions concerning the layout such as changing colors and font size in some pages; and permission to print a report of the student's performance.

The result of the evaluation by students concerning the usability of the software is presented in Table 2. Some limitations were identified by the students concerning the visualization of some items on the screen and menu; the understanding of the activity's results and a lack of discussion in the classroom concerning mistakes; slowness of the system (due to the internet connection); difficulty on the part of students to identify the activity step they were working on; understanding of the levels of relationship presented in the activity; and the fact that the exercises sometimes became repetitive and tiresome.

Some positive aspects indicated by the students included: the positive relevance of the software to improving NDx learning and that the software shows the student's level of knowledge of diagnoses; the fact the software helps students to think and reflect on the correct diagnosis, thus collaborating in the education of more competent nurses. They also suggested broadening the program to include interventions and nursing outcomes.

DISCUSSION

Fuzzy logic was used to develop the Fuzzy Kitten software in accordance with the theory's characteristics, such as the efficacious applicability of imprecise linguistic terms, and permitting the use of natural language to enter data; the possibility of evaluating the students' performance and compare it to that of experts (gold standard); and adapt the theory to the diagnostic process, in which interpretation is not always precise since it is a subjective evaluation.

Comparison of values determined by experts with those determined by students allows the analysis of student knowledge concerning the diagnostic process. The diagnostic process is seen as the establishment of the degree of relationship among DC/RF and NDx, degree in which DC/RF are present in the clinical case, and the selection of diagnostics present in the clinical case. As well as on which part of the diagnostic process the student should focus most to improve his/her performance. The scores suggest in which part of the decision-making process of the diagnosis the student needs greater skills.

The activity proposed in the Fuzzy Kitten software is intended to encourage the students' metacognition capacity, that is, make them reflect on their thinking or the steps that led them to a given diagnostic decision⁽²⁾. The reflection of students concerning their performance compared to that of experts incites them to investigate and improve their rationale for NDx. The software can help nursing professors because it is an objective method to evaluate students concerning their knowledge of nursing phenomena and of how much each DC/RF contributes to the determination of diagnoses.

The quality of a software program is a set of properties to be satisfied to a given degree, so that the software meets the needs of its users⁽¹⁵⁾. The experts determined that the Fuzzy Kitten software met its objective and satisfactorily met needs because it has the structure required to deliver what it has proposed to do, is user-friendly and complies with specifications and use standards.

In relation to the doubts presented by the experts when evaluating the software, some of them had not taken into account information present in the manual and sometimes they did not even answer the item and selected the option it does not apply.

In the evaluation of usability, the software was considered great, very good or good by 96.2% of the students. The evaluation of the software's usability showed that from the students' perspective, the activity was beneficial to learning NDx.

Some suggestions of the experts and students can be implemented in the software but first the impact of such changes need to be evaluated in terms of whether they will improve the software's use.

We intend to increase the number of clinical cases available in the software, including clinical situations and diagnoses of varied specialties. For that, one has to validate nursing diagnoses in new clinical cases and establish the degree of relationship present in the nursing diagnoses, defining characteristics and risk factors and the clinical cases.

The software can benefit nursing students' learning, allow a more objective evaluation of professors, generate new research concerning diagnostic accuracy and the students' rationale processes in determining a diagnosis, and be adapted to new possible uses, considering that it is a simple model.

CONCLUSION

We hold that the software achieved its initial objective to make the evaluation of diagnostic accuracy of students measurable, to facilitate learning concerning nursing diagnoses and allow students to more objectively assess their knowledge concerning NDx.

We expect the software to become an educational tool in teaching NDx that contributes to the education of nurses in making nursing diagnoses with greater accuracy.

REFERENCES

• 1. Lopes MHBM. Model for evaluation of diagnosis accuracy based on fuzzy logic. In: Poster presented at NANDA International Conference November; 2008 Nov.13-15; Miami (EUA).
• 2. Lunney M. Pensamento crítico e diagnósticos de enfermagem: estudos de caso e análises. Porto Alegre: Artmed; 2004.
• 3. Muller-Staub M, Lavin MA, Needham I, Achterberg TV. Nursing diagnoses, interventions and outcomes - application and impact on nursing practice: systematic review. J Adv Nurs. 2006; 56(5):514-31.
• 4. Rolfe G. Science, abduction and the fuzzy nurse: an exploration of expertise. J Adv Nurs. 1997;25(5):1070-5.
• 5. Zadeh LA. Fuzzy sets. Inform Contr. 1965;8:338-53.
• 6. Pedrycz W, Gomide F. An introduction to fuzzy sets: analysis and design. London: MIT Press; 1998.
• 7. Pedrycz W, Gomide F. Fuzzy systems engineering: toward human-centric computing. New Jersey: John Wiley & Sons; 2007.
• ⁸
  North American Nursing Diagnosis Association (NANDA). Diagnósticos de enfermagem da NANDA: definições e classificação - 2007-2008. Porto Alegre: Artmed; 2008.
• ⁹
  Associação Brasileira de Normas Técnicas (ABNT). NBR ISO/IEC - 14598-6. Engenharia de software: avaliação de produto: parte 6: documentação de módulos de avaliação. Rio de Janeiro: ABNT; 2004.
• 10. Sperandio DJ. A tecnologia computacional móvel na sistematização da assistência de enfermagem: avaliação de um software-protótipo [tese doutorado]. Ribeirão Preto: Escola de Enfermagem de Ribeirão Preto, Universidade de São Paulo; 2008.
• ¹¹
  Associação Brasileira de Normas Técnicas (ABNT). NBR ISO/IEC 9126-1. Engenharia de software: qualidade de produto: parte 1: modelo de qualidade. Rio de Janeiro: ABNT; 2003.
• 12. Borges PRT. Qualidade de software educacional: critérios para validação de treinamentos multimídia utilizados em educação à distância. Tecnol Educ. 1998;26(140):11-7.
• 13. Perfeito JAJ, Forte V, Giudici R, Succi JE, Lee JM, Sigulem D. Desenvolvimento e avaliação de um programa multimídia de computador para ensino de drenagem pleural. J Bras Pneumol. 2008;34(7):437-44.
• 14. Matos FGOA, Cruz DALM. Development of an instrument to evaluate diagnosis accuracy. Rev Esc Enferm USP [Internet]. 2009 [cited 2010 May 20];43(n.esp):1088-97. Available from: http://www.scielo.br/pdf/reeusp/v43nspe/en_a13v43ns.pdf
• 15. Rocha ARC. Análise e projeto estruturado de sistemas. Rio de Janeiro: Campus; 1987.

^*

Desarrollo y evaluación de un software que verifica la exactitud diagnóstica

Correspondence addressed to:

Publication Dates

• Publication in this collection
  20 Mar 2012
• Date of issue
  Feb 2012

History

• Received
  16 Sept 2010
• Accepted
  07 July 2011