Evaluation of the functional performance and technical quality of an Electronic Documentation System of the Nursing Process1

Objective: To evaluate the functional performance and the technical quality of the Electronic Documentation System of the Nursing Process of the Teaching Hospital of the University of São Paulo. Method: exploratory-descriptive study. The Quality Model of regulatory standard 25010 and the Evaluation Process defined under regulatory standard 25040, both of the International Organization for Standardization/International Electrotechnical Commission. The quality characteristics evaluated were: functional suitability, reliability, usability, performance efficiency, compatibility, security, maintainability and portability. The sample was made up of 37 evaluators. Results: in the evaluation of the specialists in information technology, only the characteristic of usability obtained a rate of positive responses of less than 70%. For the nurse lecturers, all the quality characteristics obtained a rate of positive responses of over 70%. The staff nurses of the medical and surgical clinics with experience in using the system) and staff nurses from other units of the hospital and from other health institutions (without experience in using the system) obtained rates of positive responses of more than 70% referent to the functional suitability, usability, and security. However, performance efficiency, reliability and compatibility all obtained rates below the parameter established. Conclusion: the software achieved rates of positive responses of over 70% for the majority of the quality characteristics evaluated.


Introduction
The use of computing systems and software has grown rapidly in all sectors of activity of society, reaching an ever greater number of users. Software is used in the areas of education, entertainment, transport, communication, the financial system, the environment, industry, business, medicine and many others.
The quality of the health information systems is a worldwide concern, and a series of initiatives is being undertaken in countries including the United States, Canada and the United Kingdom, so as to promote security in the design, acquisition and implantation of information technology in the health area (1) .
In Brazil, the implantation of computational tools integrated with the Electronic Health Record (EHR) for documenting the Nursing Process (NP) has been a gradual process, and is found in different stages of implementation.
In this context, the Teaching Hospital of the  (2) .
It is considered that the development of this software represents an advance for nursing through its use of standardized systems of language which serve to demonstrate nursing's contribution to individuals' health, allowing the measuring both of clinical efficacy and the cost of the nursing care.
Nevertheless, ensuring this system's quality is an important challenge and goal due to the responsibility for users and patients. It is believed that a new technology can bring profound transformations, and that a system can be considered successful when it meets the users' needs, is easy to use, is not prone to failure, and changes things for the better (3) .
The PROCEnf-USP ® was evaluated during the implementation phase, and the nurses judged items such as visual comfort and the handling of the system, documentation, information and content. The nurses evaluated the system positively, mainly because it contributes to the nurse's clinical reasoning and because it relates diagnoses, results and interventions (4) .
In the international scenario, the experience of administrators, nursing managers, nurses and other users with the implementation of EHR evidenced the achievement of operational advances through greater access to information, increasing the accuracy of documentation, implementation of evidence-based practice and cost reduction, as well as improvements in the quality of the care and greater staff satisfaction.
Emphasis is also placed on the positive evaluation from the administrators in relation to the return on the investments (5) .
One study on the evaluation of changes in the quality of the processing of information in nursing, immediately prior to, and one year after, the introduction of a computerized nursing system concluded that there had been significant improvement in the quality of the nursing documentation, as well as support during the patient anamnesis and care planning. The authors observed a lack of clarity in relation to the time spent on the electronic nursing documentation and its impact on patient care (6) .
One of the approaches for evaluating information technology is the use of quality standards devised and revised by the ISO (International Organization for Standardization) and the IEC (International Electrotechnical Commission). The ABNT (Brazilian

Association of Technical Standards) series -Brazilian
Regulatory Standard (NBR) ISO/IEC 9126 (7) and ISO/ IEC 14598 (8) deal with the quality of software products.
In 2011, these regulatory standards were restructured and came to be termed ISO/IEC 25010 -System and Software engineering -(SQuaRE) -System and software quality models (9) and ISO/IEC 25040 System and Software engineering -(SQuaRE) -Evaluation process (10) .
In Brazil, these norms were used for evaluating the technical quality and functional performance of a prototype-software for documenting the NP, contributing to identifying the quality of the system and elucidating the needs for improvements so as to optimize the use of the tool (3) .
In the light of the above, it was considered fundamental to undertake an evaluation of the PROCEnf-USP® system following its implantation, with a view to measuring the performance of the system and the user satisfaction. A further aim was to identify possible technical shortcomings and limitations such that improvements could be made in the final product.
As a result, this study aimed to evaluate the functional performance and technical quality of the PROCEnf-USP® system, using the ISO/IEC 25010 Product Quality Model (9) and the ISO/IEC 25040 Evaluation Process (10) . The first stage of the Evaluation Process was the choice of the ISO/IEC 25010 Product Quality Model (9) , which specifies eight quality characteristics, subdivided into sub-characteristics, as follows: functional suitability (functional completeness, functional correctness, and functional appropriateness); performance efficiency (time-behavior, resource utilization and capacity); compatibility (co-existence and interoperability); usability (appropriateness recognizability, learnability, user error protection, operability, user interface aesthetics and accessibility); reliability (maturity, fault tolerance, recoverability and availability); security (confidentiality, integrity, non-repudiation, accountability and authenticity); maintainability: (analyzability, modifiability, modularity, reusability, and testability); portability (adaptability, installability and replaceability).
These two last characteristics were evaluated only by the specialists in information technology.
The quality characteristics and sub-characteristics were evaluated through key questions, adapted from the evaluation instrument used by Sperandio (3) .
Following the updating of the standard, new characteristics were added and others received more accurate names. Due to this change, it was necessary to add new questions to the evaluation instruments.
For this, the SBIS/CFM Certification Manual was used, principally for the questions regarding the characteristic of security.
In the second stage of the evaluation, the scoring levels and the criteria for judgment were defined. For each question from the evaluation instruments, the specialists attributed the following scores: A (Agree); D (Disagree), NA (Not Applicable). Score A signifies that the PROCEnf-USP ® meets the requirement; D, that it does not meet the requirement, and NA corresponds to the attribute that it was either not evaluated or was considered not to be applicable to the software.
Those items evaluated as Disagree were explained in writing by the evaluators, so as to identify the needs for improvement in the PROCEnf-USP ® .
In order to obtain the values of the quality characteristics, following application of the evaluation instruments, the formula for calculating the characteristics and sub-characteristics proposed in ABNT NBR ISO/IEC 14598-6 Annex C (Informational) (11) was adapted: Vc = ∑ Vsc/nsc and Vsc = ∑ m/ (n-nd).
In which: Vc is the value of the characteristic measured; Vsc is the value of the sub-characteristic measured; nsc is the number of sub-characteristics; m is 1, if the response is positive, otherwise being 0; n is the total 245 www.eerp.usp.br/rlae Oliveira NB, Peres HHC.
number of measurements; nd is the number of questions discarded.
Based in this formula for calculating characteristics, the 'Not Applicable' responses were discarded, as the specialists were not able to evaluate these, either as a result of lack of resources, lack of information, or even lack of specific knowledge on the subject addressed.
This type of response is not scored and cannot impair the evaluation of the product.
The tabulation of the data and the calculation of the characteristics were undertaken using the Excel ® tool.
The criteria for judging the results obtained were based in the evaluation scale for sub-characteristics proposed in ABNT NBR ISO/IEC 14598-6 Annex C (Informational) (11) , adapted by Sperandio (3) ,  In Table 5, as described in the methodology, the responses of 'Not Applicable' were discarded, as they

Results
were not applicable to the system or could not be
In order to prioritize improvements in the PROCEnf-USP ® , the frequency of the items for which the response was 'Disagree' was analyzed, and it was observed that they were related to the characteristics of usability, performance efficiency, and reliability.  (14) .
Studies which use methods of heuristic evaluation for the analysis of the electronic nursing records systems, with standardized language, demonstrated that these systems have problems of usability, with negative effects for the nurse's documentation practices and the nursing work (15) .
Furthermore, in relation to usability, the specific methods for evaluating effectiveness, efficiency and satisfaction have identified software problems which can lead to operational inefficiencies and errors in coding data, which can bring serious consequences for health (16) .
The results of the characteristic of performance efficiency demonstrated that the system's users desire software which is faster in its response time and data processing. The system must allow the nurses to undertake routine tasks with a minimum number In this regard, it is appropriate to consider that the process of implementing and maintaining a computerized system involves costs and requires the directing of concentrated efforts, involving structural, processual and financial variables. Besides this, it requires the Taking into account that the significant use of electronic records in health has become ubiquitous in the vocabulary of information technology in health, and that information has to flow, interoperability between the systems is essential. It makes the health community more integrated, allowing data to be shared among the various health professionals, the patients and the family, and also allows the health providers to analyze and exchange data between organizations (17) .
The characteristics of maintainability and portability reached the necessary percentage in the characteristics evaluated, although a high percentage of responses of 'Not Applicable' was observed. This result was similar to those found in the study undertaken by Sperandio (3) in these two characteristics.
Compatibility also obtained many responses for 'Not Applicable', except in the evaluation of the nurses from the Medical and Surgical clinics. This type of response is little desirable, as it shows that the resources made available were insufficient for undertaking the evaluation, this being a limitation of this study.

Conclusion
The undertaking of this study made it possible to evaluate the technical quality and functional performance of the PROCEnf-USP ® software, and the results obtained demonstrated the quality of the system in each attribute considered. It is concluded that the PROCEnf-USP® system stood out in five quality characteristics: functional suitability, usability, security, maintainability and portability.
The process of improving the system is continuous, with proposals for implantation in the care units of the hospital, integration with the administrative data from human resources and quality indicators, interoperability with materials systems and patients and tests systems, and the computerized dimensioning of nursing professionals, among others. It is believed that in the near future, standardized electronic documentation will allow remote and simultaneous access to data between health professionals and different organizations. This fact will improve the care provided to the patient and assist in the development of attendance protocols and research.
In this context, this study will contribute to the improvement of the systems developed for documenting the NP. This study is not terminated in this research report, but opens a path for further studies involving nurses in the evaluation process for software in the area of nursing.