Oncoaudit: development and evaluation of an application for nurse auditors

Abstracts

Objective:

To develop a web and mobile device application to search for chemotherapy drugs to support nursing audits of hospital bills and to evaluate user satisfaction and tool usability.

Methods:

Research of technological production for development of an application for web and mobile technology. The product was evaluated by nurse auditors using the System Usability Scale questionnaire. It was also evalutated by health informactics professionals using Nielsen's heuristics.

Results:

The application is available at http://telemedicina6.unifesp.br/projeto/oncoaudit. The mobile version can be is accessed at http://play.google.com/intl/pt-BR/about/index.html. Nurse evalaution indicated that the web and mobile versions addressed user needs. In the usability evaluation, 14 problems were identified in the mobile version and eight in the web system. Implementation of improvements according to the evaluation findings were made in both versions.

Conclusion:

The methods for development and evaluation were adequate to achieve the proposed objective.

Nursing audit; Nursing informatics; Mobile applications; Medical informatics; Pharmaceutical preparations

Objetivo:

Desenvolver aplicativo de consulta de medicamentos quimioterápicos para sistema web e dispositivo móvel para auxiliar na auditoria em enfermagem de contas hospitalares e avaliar quanto a satisfação do usuário e usabilidade.

Métodos:

Pesquisa de produção tecnológica contendo desenvolvimento de aplicativo web e para tecnologia móvel. O produto foi avaliado quanto à satisfação por enfermeiros auditores utilizando o questionário System Usability Scale (SUS) e quanto à usabilidade pelas heurísticas de Nielsen, por profissionais de informática em saúde.

Resultados:

O aplicativo esta disponível no http://telemedicina6.unifesp.br/projeto/oncoaudit. O aplicativo móvel pode ser acessado em http://play.google.com/intl/pt-BR/about/index.html. A avaliação pelos enfermeiros indicou que o aplicativo web e móvel estão de acordo com as necessidades dos usuários. Na avaliação de usabilidade foram identificados 14 problemas no aplicativo móvel e oito no sistema web, gerando modificações am ambos.

Conclusão:

Os métodos escolhidos para desenvolvimento e avaliação mostraram-se satisfatórios para atingir os objetivos propostos.

Auditoria de enfermagem; Informática em enfermagem; Aplicativos móveis; Informática em saúde; Preparações farmacêuticas

Introduction

The growth of the supplemental health segment and the widening of care covered by the National Regulatory Agency for Private Health Insurance and Plans (ANS) led health insurances companies to develop strategies to reduce costs. In this scenario, nurses faced a new area of responsibility: serving as account auditors to verify the correspondence between care delivery and amount of transferred funds.

Auditing was introduced in the United States in 1918 to verify the quality of obstetric care, followed by other health sectors. Audits can be classified as (a) audits of structure, which monitor the structure or location of care; (b) audits of process, which measure how care is delivered; and (c) audits of results, which provide indicators of care quality. Audits occur retrospectively, i.e. after hospital discharge or simultaneously, while care is being delivered to the patient.

In nursing, audits entail the systematic evaluation of the quality of care delivery, verification of compatibility, and evaluation of number of procedures conducted and items in the hospital bill to assure adequate payment. This activity is exclusive of the nurse as stated by resolution no. 266/2001 of Brazilian Federal Nursing Council (COFEN).⁽¹⁾

In a hospital bill audit, the nurse needs to verify whether drugs listed in the bill are consistent with medical prescriptions and if they were charged according to the contracted plan. In practice, pharmaceutical guides are used to verify indication, posology, stability, administration route, and general guidance on the product by the manufacturer. The main references are the Dicionário de Especialidades Farmacêuticas (DEF [pharmaceutical dictionary]) and websites specializing in pharmaceutical guides, such as the Brazilian Health Surveillance Agency (Anvisa) electronic pharmaceutical guides list.⁽²⁾ Such activity is time consuming and requires completely reading the pharmaceutical guides, and sometimes the access to the Anvisa electronic insert list depends on the available technology, the subscription to the electronic list, and assurance that information is complete and updated.

Methods

This study is a development applied research of an application for web and mobile technology used to search chemotherapeutic drugs in hospital bill auditing.

Software development was based on system development life cycle using the prototyping concept. The phases were (a) communication, which consisted of searching the requisites for the software; (b) planning, which entailed describing the resources that would be used and the schedule to be followed; (c) designing, comprising the model to be executed in agreement with the requisites identified; (d) construction, which combined code generation and tests to reveal errors; and (e) implementation, during which the product was analyzed and assessed.

To develop the web environment it was used Drupal, a content management system. Information on each drug were added to a database created by MySQL, a database management system.⁽³⁾

The mobile application was developed for the Android plataform.⁽⁴⁾ This platform was chosen because of the ease of the Java library available via Google (the Accessory Development Kit). One of the predetermind parameters was the use of free software to avoid costs that could interfere in construction of the system. The interface was based on widely used models of eletronic pharmaceutical guides (already mentioned in this paper). In addition, it was used other well-known materials, such the Brazilian Clinical Oncology Manual and Epocrates.^(5,6) Javascript technology was applied for its feasibility, portability, easiness of use, safety, scalability, and capacity for creation of components compatible to web enviroment; this enabled the use of different platforms, such as Windows, Linux, and Unix.

To address the second objective of this study, the application was evaluated in two stages by two different groups: nurse auditors and health informatics professionals. The nurse auditors met the following inclusion criteria: at least 5 years of experience as a nurse auditor and experience in audit bills at oncology centers. Health informatics professionals invited had to have, preferably, a specialization, a master's or doctorate degree in the area, and basic knowledge of Nielsen's heuristic.

To evaluate user satisfaction of nurse auditors, it was applied the System Usability Scale (SUS) questionnaire, which is widely accepted for its trustworthiness and validity.⁽⁷⁾ This questionnaire contains a simple scale of ten items, enabling the evaluation of subjective perceptions. A Likert scale was used, with values ranging from 1 (totally disagree) to 5 (total agree); 3 indicated a neutral response. To calculate the SUS score, the contribution score of each item was summed. For items 1, 3, 5, 7, and 9, the score is the position of the scale minus 1; for items 4, 6, 8, and 10, the contribution is 5 discounting the scale position. The total sum of 10 questions, multiplied by 2.5 was used to obtain the global value of system usability. The SUS score ranges from 0 to 100; scores lower than 51 were considered poor, those higher than 71 were good, those higher than 86 were excellent, and those higher than 91 were the best possible scores.⁽⁸⁾ This questionnaire has also been applied in other studies, a factor that contributed to be selected.⁽⁹⁾

It was decided to use convenience sample where 29 nurse auditors were invited; of these, 10 accepted the invitation and participated in the ealuation. To perform an audit, nurses received a hypothetical case of a hospital bill from an oncology provider. Based on this case, the evaluator had to search in the Oncoaudit system and then complete the SUS questionnaire.

In the usability evaluation with health informatics professionals, it was decided to apply Nielsen's heuristic as the instrument. This method consisted of evaluator analysis of the interface and expression of the evaluator's option. This instrument has 10 general designed principles called "heuristics": (1) visibility of system status, (2) match between system and the real world, (3) user control and freedom, (4) consistency and standards, (5) error prevention, (6) recognition rather than recall, (7) flexibility and efficiency of use, (8) aesthetics and minimalist design, (9) helping users recognize, diagnose, and recover from errors, and (10) help support and documentation.⁽¹⁰⁾

First, the application was presented to professionals, and the evaluator inspected the application, using the heuristics as a guide to identify possible problems. The next step was to classify the severity of the problem using a scale from 0 to 4, where 0 = unimportant (did not affect the interface operation); 1 = aesthetic (no immediate need for solution); 2 = simple (low-priority problem that can be corrected); 3 = severe (high-priority problem that must be corrected); and 4 = catastrophic (very severe and must be corrected right away).⁽¹¹⁾

It was used a routine model to assess Nielsen's heuristic that has been employed in another Brazilian study.⁽¹²⁾ Due to the best cost/benefit ratio that is achieved when three to five individuals conduct an asssement, it was invited three evaluators. Evaluations occured from May to June 2013.

Development of this study followed national and international ethical and legal aspects of research on human subjects.

Results

Drugs selection was based on two pharmaceutical guides and entailed 146 drugs. A total of 30 drugs were excluded because they were duplicates or noncommercialized. After assembly of the final list, pharmaceutical companies and websites were identified. This method enabled us to include more than 68 categories of drugs, yielding 184 drugs to form the database.

Oncoaudit was made available at http://telemedicina6.unifesp.br/projeto/oncoaudit/. To access the application, users must first register. After login, the first page lists drugs in alphabetic order. Upon clicking on a drug name, information on that drug appears on a different page organized by topics, as shown in figure 1.

Figure 1
Display of selected drug, organized by topic

The following information was shown for each drug: brand name, generic name, indication, posology, compatiblity with bottles and equiment, stability, diluent, final volume and time of infusion, incompatibility with solutions, route of administration, URL for the pharmaceutical company, references with additional information about the pharmaceutical guide; distributor of the drug, pharmacologic group, pharmaceutical guide in PDF format, and date of registration and updating data. It is important to mention that of 184 drugs, nine were not included in table format because of the length of the text. In such cases, it was opted to include the information "see full prescribing information".

The mobile application uses the MySQL database manager, and is update by synchronization from the web system. The objective of this mobile application was to offer an interface for searching for, but not registering, data. Since all data were available through the web service, all communication of the application with data is done from web services, including login, information on drugs, synchronization, and archives of pharmaceutical guide. Web services were implemented directly in the web system using a plugin by Drupal.

The application stored all information of the pharmaceutical guide, including PDFs, in order to enable use even when the mobile device was not connected to the Internet. Data are updated automatically when synchronization was last performed 7 days previously or manually with use of an icon available on the application's homepage. However, it is necessary to stay connected to the Internet upon first access to perform the initial synchronization with data for the drug pharmaceutical guide and storage data of local login. Synchronization after login ensures the use of as little space as possible in the memory of the device; as a result less time is needed for downloads. In addition, it also enables users to access the application even if they are not connected in the Internet.

To access Oncoaudit on a mobile device (Figure 2), the user must first register in the web system through the link "click here". In this way, the user will know the web system and the account information will be stored.

Figure 2
Homepage and page with links available in the mobile device

The four icons developed have the following functions: (1) "look for updates", used to update the database manually, when new drugs are added in the website; (2) "contact", used to ask questions, request information, and provide suggestions; (3) "about", describing the application, its objective, and information about its authors; and (4) "drugs list", providing access to the complete drug database.

Five nurse auditors evaluated the mobile application, and the other five nurses evaluated the web system. Evaluation with the SUS questionnaire showed that the average SUS score was 90 ± 5 for the mobile application and 97 ± 5 for the web system. Nurses considered the mobile application easy to use, useful, innovative, and complete and felt that it provided important information to help them audit bills that referred to antineoplastic drugs. One of the nurses praised the topics' "stability", "pharmacological group", and "compability with bottles and equipment" considering the importance in practice. They also stated that the web system was easy to use and found that information provided is necessary for practice, allowed them to optimize reading of pharmaceutical guide as well trustable. They also suggested that other drugs should be included and the application should be used not only by auditors but also as an instrument for teaching and updating.

The three health informatics professionals assessed usability in both the mobile and web environments. Considering total evaluation in both products, they found 14 problems in the mobile version according to Nielsen's heuristic, whereas the web version showed eight problems. In both versions, only two things were considered catastrophic; two problems were considered severe in web system and four in the mobile version.

The problem for the mobile version that was classified as catastrophic was comprised in the first heuristic: visiblity of system status. The first evaluator observed a delay of the mobile version's database with the web system for synchronization of updates. The other evaluator also perceived the same problem but classified the problem as severe. The third evaluator identified another problem that was classified as severe in the heuristic principle "flexibility and efficiency of use" because when the user search for the drug in the specific field, the application did not find those with accent marks. Thus, the evaluator suggested that words with accent marks not be differentiated from words without accent marks.

In evaluation of the Oncoaudit in the web system, only one problem was classified as catastrophic and two as severe. The catastrophic problem concerned error prevention, in which the first evaluator verified that in pharmacological group locking could occur during the search. The two severe problems were related to the heuristic principle "flexibility and efficiency of use" in which two evaluators perceived that two search filters - brand name and generic name - were unnecessary; the suggestion solution was to include a single filter to find this information. In the nurses' evaluation, the nurses also suggested including a single filter to find brand name and generic names.

Discussion

Computers were introduced to nursing professionals more than 40 years ago, and they have been used in decision making systems, guidance of patient care, teaching and training, registration of nursing processes in hospital information systems, and electronic health records. In nurse audit practice, although diffusion is restricted, the informatics have been used to codify physician fees, preanalysis of hospital bill by health providers, and analysis of auditing in relation to the electronic health record.

Information technology and communication must be used to improve the professional development, decision support, and representation of resources that optimize the care process and evaluation, assuming that those who have adequate and updated information at the point of care are better able to make decisions.⁽¹³⁾

The evaluators in this study suggested that the search function should have a single filter to find both brand and generic names. However, a nurse auditor will not always know if the drug is presented with its generic or brand name.

We believe that categorizing information according to topic is faster and more intuitive; this was confirmed by results of evaluation conducted by nurse auditors. Still, having correct information on drugs is fundamental. When some information is omitted, doubts can appear that compromise the result of the analysis. After rigorous reading of pharmaceutical guide and inclusion of the content in the topics, we observed that several drugs with the same active principle had divergent information. Such a discrepancy is relevant during auditing practice and brings several conclusions in the process.

Nurse auditing in oncology is an area that often causes doubts, and few nurses identify them. The technical knowledge is imperative for releasing or auditing the charging process in oncology. Previous studies observed the uniformity and deficiency in information in pharmaceutical guide lists about the same drug, such as chemical and pharmacological characteristics, indications, contraindications, precautions and advertencies, drug interactions in adverse effects, posology, and overdosage.⁽¹⁴⁾

The standardization of drug information with the same active principle was not completed, but pharmaceutical guide constitute an important information source in Brazil that has been changed several times: between 1946 and 2006 the number of mandatory items increased and the level of description in pharmaceutical guide increased.⁽¹⁵⁾

The evaluation was conducted using the SUS questionnaire, which was satisfactory; this finding agrees with other studies that also applied the same tool, such as evaluation of management software of diabetes based on Internet technology and a web-based tutorial for parents of children with autism,⁽¹⁶⁾ as well as a system of information exchange in primary health care.⁽¹⁷⁾

In an evaluation using Nielsen's heuristic, it was understood that this tool is a good alternative to test interactive health websites in settings where time and resourses are limited. Suggestions and opinions given by nurses audit and informatics health professionals helped improve the application through their useful and practical evaluations.

Technological advances enable nurses to direct their professional destiny and to adapt technology resources based on practice; they also help nurses to see urgent trends in the health area as a challenge and a single opportunity for careers growth. There are new tools, new areas, and new activites demading specialists in any country, and there are many opportunities for those who decide to incorporate technological information into daily practice.

With this research and the development of this application, ir was also expect to incentive the interest of nurse auditors in developing studies to improve the technical knowledge in nursing audit in consonance with technology for professional practice.

Conclusion

It was developed an application containing information to support auditing of drugs in hospital bills. This tool was evaluated with regard to user satisfaction and usability, which helped promote improvements in software before it becomes available to the public. Even with the limited number of evaluators, it was clear that Oncoaudit can be used in practice for drugs audit. This application can make the auditing process faster and complete. In addition, the study results suggest that the application can have high impact if more pharmacological groups are included.

Acknowledgements

Dr. Marin acknowledge the support provided by CNPq 477394-2011-6 and 301735/2009-3. We also thank the partial support of the grant 5D43TW007015-08 from Fogarty International Center and the National Library of Drug, National Institutes of Health.

Corresponding author: Heimar de Fatima Marin, Napoleão de Barros street, 754, São Paulo, SP, Brazil. Zip Code: 04024-002. [e-mail]

Conflicts of interest: none to report.

• Collaborations
  Grossi LM contributed to the conception of the project, analysis and interpretation of the data and drafting of the manuscript. Pisa IT contributed with analysis and interpretation of the data and critical review to improve the manuscript intellectual content. Marin HF contributed to the conception of the project, analysis and interpreation of the data, reviews and drafting of the manuscript to improve its intellectual content and approval of thi final version for publication.

Referências

• ¹
  Cofen: Conselho Federal de Enfermagem. Resolução Cofen-266/2001 [Internet]. 2001 [citado 2013 Jul 13]. Disponível em: http://novo.portalcofen.gov.br/resoluo-cofen-2662001_4303.html.
  » http://novo.portalcofen.gov.br/resoluo-cofen-2662001_4303.html
• ²
  Brasil. Ministério da Saúde. Agência Nacional de Vigilância Sanitária. Bulário Eletrônico [Internet]. Brasília: Anvisa; c 2005-2009 [citado 2013 Jul 13]. Disponível em: http://www.anvisa.gov.br/datavisa/fila_bula/index.asp.
  » http://www.anvisa.gov.br/datavisa/fila_bula/index.asp
• ³
  MySQL: The world's most popular open source database [Internet]. c2013 [cited 2013 May 19]. Available from: http://www.mysql.com.
  » http://www.mysql.com
• ⁴
  Licenses. Android Open Source [Internet]. [cited 2013 May 19]. Available from: http://source.android.com/source/licenses.html.
  » http://source.android.com/source/licenses.html
• ⁵
  Manual de oncologia do Brasil. MOC 2013 [Internet]. c2012 [citado 2013 Jul 13]. Disponível em: http://mocbrasil.com.
  » http://mocbrasil.com
• ⁶
  Epocrates [Internet]. c2013 [citado 2012 Maio 19]. Disponível em: http://www.epocrates.com/
  » http://www.epocrates.com/
• ⁷
  Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. International J Hum Comput Interact. 2008;24(6):574-594.
• ⁸
  Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective ratingscale. J Usabil Stud. 2009;4(3):114-23.
• ⁹
  Tenorio JM, Sdepanian VL, Pisa IT, Cohrs FM, Marin HF. Desenvolvimento e avaliação de um protocolo eletrônico para atendimento e monitoramento do paciente com doença celíaca. Rev Inform Teór Aplic. 2011;17(2):210-20.
• ¹⁰
  Nielsen J. How to Conduct a Heuristic evaluation [Internet]. 1995 [cited 2013 Dec 19]. Available from: www.nngroup.com/articles/how-toconduct- a-heuristic-evaluation.
  » www.nngroup.com/articles/how-toconduct-a-heuristic-evaluation
• ¹¹
  Nielsen J. Severity ratings for usability problems [Internet]. 1995 [cited 2013 Dec 19]. Avaialble from: www.nngroup.com/articles/how-torate- the-severity-of-usability-problems.
  » www.nngroup.com/articles/how-torate-the-severity-of-usability-problems
• ¹²
  Gonçalves LL. EditWeb: mecanismos de autoria assistida de páginas para ambiente de EAD via web visando usabilidade e acessibilidade. [Internet]. Porto Alegre (RS). Universidade Federal do Rio Grande do Sul; 2004 [citado 2013 Nov 28]. Disponível em:http://www.lume.ufrgs.br/handle/10183/8694.
  » http://www.lume.ufrgs.br/handle/10183/8694
• ¹³
  Marin HF. Sistemas de Informação em saúde: considerações gerais. J Health Inform. 2010;2(1):20-4.
• ¹⁴
  Gonçalves SA, Melo G, Tokarski MH, Branco AB. Bulas de medicamentos como instrumento de informação técnico-científica. Rev Saúde Pública. 2002; 36(1):33-9.
• ¹⁵
  Caldeira TR, Neves ER, Perini E. Evolução histórica das bulas de medicamentos no Brasil. Cad Saúde Pública. 2008;24(4):737-43.
• ¹⁶
  Kobak KA, Stone WL, Wallace E, Warren Z, Swanson A, Robson K. A web-based tutorial for parents of young children with autism: results from a pilot study. Telemed J E Health. 2011;17(10):804-8.
• ¹⁷
  Haarbrandt B, Scwartze J, Gusew N, Seidel C, Kleinschmidt T, Haux R. Primary care provider's acceptance of health information exchange utilizing IHE XDS. Stud Health Technol Inform. 2013;192:106-8.

Publication Dates

• Publication in this collection
  Mar-Apr 2014

History

• Received
  07 Mar 2014
• Accepted
  18 Mar 2014