APPLICABILITY OF AN ASSESSMENT MODEL FOR HEALTHCARE INFORMATION SYSTEMS IN A PUBLIC HOSPITAL

ABSTRACT

Assessment processes are essential to guarantee quality and continuous improvement of software in healthcare, as they measure software attributes in their lifecycle, verify the degree of alignment between the software and its objectives and identify unpredicted events. This article analyses the use of an assessment model based on software metrics for three healthcare information systems from a public hospital that provides secondary and tertiary care in the region of Ribeirão Preto. Compliance with the metrics was investigated using questionnaires in guided interviews of the system analysts responsible for the applications. The outcomes indicate that most of the procedures specified in the model can be adopted to assess the systems that serves the organization, particularly in the attributes of compatibility, reliability, safety, portability and usability.

Keywords:
Health information systems; Quality assessment; Software metrics

1. INTRODUCTION

The global health observatory data repository, from the WHO, with information about healthcare investments in over 190 countries, shows a rising curve of expenses per capita in health (WHO,2013). According to Newell (2011), the costs for funding healthcare have grown globally due to factors such as increased life expectancy, advances in healthcare technology and policies for universal access to healthcare, in spite of government actions to mitigate budgetary impacts, with public budget constraints, above all after the economic crises that have occurred on a global scale, since 2008. In countries where healthcare systems are private or mixed, they also try to minimize these costs, to make insurance and health plan operators economically feasible.

In healthcare organizations, healthcare information systems (HIS) add information technology and communication to address their processes (Ammenwerth et al., 2004) and integrate people, procedures and technologies to collect, store, manipulate and recover information (Wager, Lee & Glaser, 2009). They are characterized by complex and multidisciplinary deployment, produce impacts in learning and in the adaptation to organizational routine and involve several groups of stakeholders - patients, service providers, regulating agents and professionals (Fichman, Kohli& Krishnan, 2011).

Investments in information systems can constitute part of the healthcare organization policies to reduce the tension between costs and budgets, in order to improve efficiency and quality in the processes that occur in this sector. The information systems improve healthcare organization efficiency, reduce medical prescription error rates, help professionals and managers in decision-making and in preventive medicine (Hillestad et al., 2006; Ammenwerth et al., 2003) and have great potential to reduce costs and improve healthcare outcomes (Fichman, Cohli&Krisnan, 2011; Oliveira et al., 2011). Research outcomes on the assessment of information systems for the healthcare sector also show how administrators recognize the importance of information systems as critical resources and that there is great demand to align information systems to the management process (D´Souza &Sequeira, 2011).

The development and maintenance of healthcare information systems are complex activities, due to: (a) lack of standardization and interoperability difficulties between applications (Hillestad et al.,2006), (b) the interdisciplinary characteristic of healthcare that demands added knowledge from several user professionals in the construction of information systems (Fichman, Kohli& Krishnan, 2011; Carvalho&Eduardo, 1998) and (c) the fragmented nature of the healthcare sector and the difficulties to systematize processes in applications (Abouzahr&Boerma, 2005), besides the actual change in paradigm, of a reactive model, centered on the disease, to a preventive model, that makes communication flow difficult among the three levels of attention and in continuous attention (OPAS, 2011).

Within this context of complexity, assessment is an essential activity to guarantee healthcare software quality as well as its continuous enhancement. Software assessment activities measure the attributes of a system in several phases of its lifecycle, help in the optimization of outcomes, identify unpredicted events (Ammenwerth et al.,2004) and allow us to analyze the degree to which information systems address their objectives (Yusofet al., 2008).

This article specifies 42 software metrics as a technique to measure quality attributes established in a structured model guided towards healthcare information system assessment and verifies the feasibility of these metrics in applications that offer support to clinical, outpatient and administrative processes in a public hospital that serves the macro region of Ribeirão Preto.

2. THE QUALITY ASSESSMENT MODEL

The model proposed by Morais & Costa (2013), presented in Figure 1 was used as a theoretical reference. The model uses attributes from the product's quality dimension in Rule ISO/IEC 25010 (ISO, 2011a), which includes eight quality characteristics: functional supportability, performance efficiency, compatibility, usability, reliability, safety, maintainability and portability. Each characteristic is made up of a set of sub-characteristics, which are described in Attachment I.

Figure 1
Proposed assessment framework

Due to the degree of subjectivity found in the sub-characteristics - inherent to the model, as it can be applied to any software product - each attribute was associated to a set of indicators, obtained in a systematic research process in databases, which selected 32 indicators from seven relevant papers: Paiand Huang (2011), Viitanen et al.(2011), Hubner-BloderandAmmenwerth (2009), Ribière et al.(1999), Otieno et al.(2008), Anderson and Aydin(2005) and Lima et al.(2009).

Using a semantic analysis, each indicator was classified into a characteristic and a sub-characteristic of the product's quality dimension from Rule ISO/IEC 25010 (ISO, 2011a). Another 10 indicators were added by the authors based on the reading of documents ISO/IEC 25023 (ISO, 2011b) and ISO/IEC 9123-3 (ISO, 2003). The inclusion of these indicators covered all the quality dimension characteristics of the product, thus establishing the framework.

For each indicator, the model proposes assessment questions and/or software inspection procedures, applicable to different stakeholders, to objectively assess the indicator. The model includes the following as stakeholders:

• - Managers: users of the system at its strategic level;

• - Health Professionals: users of the system at tactical and operational levels;

• - Patients/users: access the system only for queries;

• - IT Professionals: compose the technical staff for the application, such as developers and/or maintainers of the information systems.

The assessment questions were elaborated and grouped into questionnaires guided towards user profiles (managers, health professionals and patients) and are specified in Morais & Costa (2013). The elaboration of questions was guided in the writing of clearly-expressed texts without ambiguities and in language that was easy to understand. Measuring scales were also developed observing clear and appropriate reading and coverage of the universe of possible answers with uniform distribution. Ordinal and interval scales were used, according to the definition from Malhotra (2006), with five response options organized in increasing order of adequacy.

The inspection procedures proposed in the model must be directed towards IT professionals who maintain applications, stakeholders who have access to information about the requirements of the systems, track record of changes, track records of defects and system failures and other relevant information to obtain the measurements predicted. These procedures must use the documental analysis, tests/simulations of software use or performance measurements/essays as ways of obtaining measurements.

Attachment II describes the detailing and specifications of the model's assessment questions and inspection procedures. The first column of Attachment II indicates the stakeholder users of the application for which the assessment questions were guided towards. The metrics specified in the last column are directed towards the application maintainers.

When including questionnaires directed to the application users and inspection procedures guided towards maintainers, one distinguishing characteristic of this model is the fact that it "listens to both sides". Many times, conflicts are established due to lack of communication (Dallavalle, 2000), the distance between the organization and the service provider (Albertin& Moura, 1995) or due to inadequate implantation processes identified by Caldas & Wood (2000), as acquisition of software without clear-cut criteria, by imposition or low involvement of the user.

3. METHOD

The work done was qualitative and it was divided into a first conceptual phase, with the specification of inspection procedures for the model used and a second empirical phase, with the investigation of compliance to the metrics obtained in the first phase for three applications in a public hospital offering regional medium and high complexity services.

The first phase of this work is classified as methodological research as it refers to an instrument that takes in the reality studied: according to Vergara (2005, p. 47), the methodological research "is associated to paths, forms, ways, procedures to reach a certain end," characterized in this study.

The specifications described in this first phase used the following documents ISO/IEC 25023 (ISO, 2011b) and ISO/IEC 9126-3 (ISO, 2003) as a basis with context adaptations for healthcare information systems. The scales for the metrics specified in the inspection procedures are valued between zero and one: the closer they are to one, the greater the compliance of the system with the specified metric.

A survey was made in the second phase of the research using interviews guided towards system analysts responsible for the applications. A questionnaire was answered in the interviews, where the interviewer asked about the feasibility of each specified metric for the application maintained by the system analyst, with the response options "feasible/unfeasible/not applicable to the context". The option "not applicable to the context" refers to cases where the interviewee assesses that the metrics proposed were incongruous with the target system. Remarks from professionals were noted, regarding feasibility or non-feasibility of the inspection procedures.

The interviews with the application maintainers were carried out in July 2013, in sessions of 90 minutes' maximum and included part of the time for suggestions given by the professionals. As a limitation, it must be observed that these studies were restricted to the analysis of procedures only for the stakeholders included in the assessment model, to verify and describe the applicability of the metrics proposed in the model. An assessment and application of instruments guided towards other stakeholders is a proposal for future research work.

4. OUTCOMES

4.1. The specifications of the software inspection procedures

Chart 1 describes the indicators and software metrics inspection procedures specified for the assessment model. The first column numbers its 38 indicators, classified in the model's characteristics and sub-characteristics. Inspection procedures were not specified for four indicators: "Extensive training is not needed to learn about the system", "The interface (screens, forms, data entry, reports or graphs) as well as all the terms and concepts used in the system are clear and have no ambiguities", "The system is easy to use, intuitive" and "The system presents a uniform and standardized interface", all of them have the usability characteristic. Psychometric assessment questions guided towards users were elaborated for these indicators, to answer how many system functions comply with the indicators, using an ordinal scale of five points: 0% (none), up to 25% (few), between 25 and 75% (about half), more than 75% (most) and 100% (all).

Chart 1
Indicators and software inspection procedures for the assessment model

Characteristics/sub-characteristics Quality Indicators	Software Metrics Specifications
Functional supportability/ Functional completeness 1. The system offers support/ helps in decision-making 2. The system obeys legal information rules (CID10, DRG, data transmission, etc.) 3. The system helps to prevent medication errors 4. The clinical documentation generated by the system is correct and complete 5. The information treated by the system addresses the users' transactional operations	ISO/IEC 25023 - Functional deployment coverage metric: NFDR= number of functions included in requirement documents with support for decisions NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC1 = 1 - (NFAI / NFDR) NFDR= number of functions related to requirement documents that demand compliance with legal information rules NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC2 = 1 - (NFAI / NFDR) NFDR= number of functions related to requirement documents that demand compliance with verification and prevention of errors in medication NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC3 = 1 - (NFAI / NFDR) NFDR= number of functions related to requirement documents that demand functionalities that include clinical documentation NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC4 = 1 - (NFAI / NFDR) NFDR=number of functions related to requirement documents referring to transactional operations NFAI = number of absent or incorrect functions, among those identified in NFDR SUPFUNC5 = 1 - (NFAI / NFDR)
Functional supportability/ Functional correctness 6. The system makes correct information available	ISO/IEC 25023 - Computational accuracy metric: NACI = number of attributes with incorrect computations in an execution verification in a certain time interval (execution simulation) NTCV = total number of attributes observed in the verification of execution SUPFUNC6 = 1 - (NACI/NTCV)
Functional supportability / Functional adequacy 7. The system integrates processes from different areas and/or departments	ISO/IEC 25023 -Functional adequacy metric: NFAI = number of system functions that do not deploy or partially deploy the automated support for integration between functional areas or departments, among those that demand this function NTFI = total number of functions that demand automated support for integration between functional or department areas, predicted in requirement documents SUPFUNC7 = 1 - (NFAI/NTFI)
Performance efficiency/ Behavior with regard to time 1. The performance of the system is satisfactory: data is processed in an acceptable period of time; the system responds quickly to entries 2. The clinical documentation generated by the system addresses the time constraints demanded 3. Authentication time for system access is adequate	ISO/IEC 25023 - Response-time and turnaround time metrics: TE: execution time of a simulation of online operation execution TME: maximum execution time of an online operation, specified in each non-functional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP1 = Arithmetic average of ATR values, obtained in simulations on online operation executions TE: execution time of a simulation of the execution of a job TME: maximum execution time for a job, specified in each non-functional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP2 = arithmetic average of ATR values TE: execution time for an authentication simulation TME: maximum execution time for an authentication operation, specified in non-functional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP3 = arithmetic average of ATR values, obtained in simulations of authentication operation execution
Performance efficiency/ Capacity 4. The system offers adequate simultaneous access, with satisfactory performance	ISO/IEC 25023 - Number of simultaneous accesses metric: NAS=verified number of simultaneous accesses by users, with performance within adequate standards NMA=maximum number of simultaneous accesses predicted in the system requirements EFDESEMP4=NAS/NMA
Compatibility/Coexistence 1. The system predicts access to its data from other systems	ISO/IEC 25023 - Coexistence availability metric: NADC: number of applications for which system makes coexistence available NARE: number of applications that require coexistence of system COMPAT1 = NADC/NARE
Compatibility/ Interoperability 2.The system can be integrated or connected to exchange information with other systems	ISO/IEC 25023 -Connectivity with external systems metric: NIOA: number of interfaces with other correctly deployed applications, to exchange data. NIR: number of interfaces required, for connectivity with other systems. COMPAT2 = NIOA/NIR
Usability/Learnability 1. The system has manuals, tutorials, documentation for training and access to data and/or help online available	ISO/IEC 25023 - user documentation completeness/help facilities metric NFDD: number of correctly described functions in the user documentation and/or in the online help functions for users NTF: total number of functions in the system accessible to users USAB1 = NFDD/NTF
Usability/Operability 2. Browsing through the system is quick and standardized 3. The system offers adequate feedback to user for tasks performed 4. The software allows adaptations to address local/specific needs, by user himself	ISO/IEC 25023 - Operational consistency metric: NFNU: number of functions with uniform/standardized browsing. NTF: total number of system functions accessible to users. USAB2 = NFNU/NTF ISO/IEC 25023 - Message clarity metric: NFMC: number of functions with messages that include easily understood terms NTF: total number of system functions accessible to users USAB3 = NFMC/NTF ISO/IEC 25023 - Customization possibility metric: NFC=number of functions that can be customized by user in operations, among those that require this resource. NTFC=number of functions that demand, in their requirements, the possibility of customization. USAB4=NFC/NTFC
Usability/ Protection from user error 5. Simple, easy and safe to correct an error (reversibility) 6. The system deploys verification of valid values in data entries 7. The system avoids incorrect operations	ISO/IEC 9126-3 - Ease to cancel metric: NFC: number of functions that allow cancellation of execution before the end. NTF: total number of functions in the system that predict cancellation of its requirements. USAB5 = NFC/NTF ISO/IEC 9126-3 - Ease to undo metric: NFD: number of functions that allow undoing of execution after ended NTF: total number of functions in the system that predict cancellation of execution in requirements USAB6 = NFD/NTF ISO/IEC 25023 -Entry verification and validation metric: NACV = number of entry attributes with verification for valid data NTAE = total number of entry attributes USAB7 = NACT / NTAE ISO/IEC 25023 - Prevention of incorrect operations metric: NFCO = number of functions that prevent execution of incorrect operations NTF = total number of system functions USAB8 = NFCO / NTF
Usability/Esthetics of user interface 8. The arrangement of the interface fields are adjustable to the user's work	ISO/IEC 25023 - Appearance customization of user interface metric: NIC: number of interfaces that can be customized in their appearance by user NTI: total number of interfaces with system user USAB9 = NIC/NTI
Usability/Accessibility 9. The system includes access facilities for users with special physical needs/per age	ISO/IEC 25023 -Physical accessibility metric: NFNE = number of functions accessible to people with special needs/elderly NTF = total number of functions in system USAB10 = NFNE / NTF
Reliability/Maturity 1. The system presents low rates of software maintenance calls 2. The system is reliable, stable and errors do not occur when it is being used 3. The corrections, improvements or updating of version do not cause instability in the system nor demand effort or excessive time	ISO/IEC 25023 -MTBF - Media Time Between Failures metric: Observing the MTBF of the system as from its implantation (or the implantation of its latest version), in regular periods (half a year, one year), assess the evolution of metric CONF1= 0 if MTBF decreasing, 0.5 if MTBF constant and 1.0 if MTBF increasing ISO/IEC 9126-3 - Failure detection metric: CONF2=quantity of failures detected in last review/estimated failures Maintenance re-incidence metric (metric proposed by author): MR= for a given period, compile number of maintenance calls arising from prior maintenance calls, that generate side effects (and as a consequence make the system unstable) NTM= total number of maintenance calls in period CONF3= MR/NTM
Reliability/ Availability 4. The system is always available, accessible to the user	ISO/IEC 25023 - Rate of system in production metric: HSP = Hours in which system was in production, available to the user in a pre-defined period (last month, quarter, for example) NTHD = Hours in which system should be in production in a pre-defined period CONF4 = HSP / NTHD
Reliability/ Tolerance to failure 5. The system has resources to store redundant data	ISO/IEC 25023 - Component redundancy metric: NCR = number of components installed redundantly to mitigate failures NCRI = number of redundant components installed predicted in system requirements CONF5 = NCR/NCRI
Reliability/Recoverability 6. The system present level of data loss and efficient restoration mechanisms	ISO/IEC 9126-3 - Restorability metric: NRR = number of restorations required, in a given time interval NRRS = number of successful restorations required, in a given time interval CONF6 = NRR/NRRS
Safety/ No questioning 1. The system includes a digital signature	ISO/IEC 25023 - Digital signature use metric: NEAD = number of events processed using a digital signature NTAD = total number of events that demand a digital signature SEG1 = NEAD / NTAD
Safety/ Confidentiality, Authentication and Integrity 2. There is no risk of unauthorized access to the system information	ISO/IEC 25023 - Access controllability metric: TCAI = number of types of access controls (verification of illegal operation) correctly deployed and verified TCAP = number of types of access controls predicted in system requirements SEG2 = TCAI / TCAP ISO/IEC 25023 - Data encryption metric: NIDCE = number of data items correctly encrypted/decrypted NIEP = number of data items with encryption predicted in system requirements SEG3 = NIDCE / NIEP
Safety/ Accountability 3. The system has auditing and /or tracing mechanisms	ISO/IEC 25023 - Access auditability metric: NFRL = number of functions with log register, with information about access and/or modification of data made by a user NTF = total number of functions in system SEG4 = NFRL / NTF
Maintainability / Modularity and Modifiability 1. Software integrates (easily) with new /systems	ISO/IEC 25023 - Condensability metric: NCNA= number of components not affected by alterations in other components NTC= total number of components MANUT1 = NCNA/NTC ISO/IEC 25023 - Modification success rate metric: PRAM = number of problems/complaints before maintenance PRDM = number of problems/complaints after (same) maintenance MANUT2 = (PRAM-PRDM)/PRAM Note: The system calculates an average for the set of modifications over a given period
Maintainability/ Reusability 2. The system has reusable software components	ISO/IEC 25023 - Degree of reusability metric: NCRS = number of reusable software components/artefacts, that can be used in more than one system or used to build other systems NTCB = total number of components/software artefacts that can be reused in object library reused in development environment of software product MANUT3 = NCRS / NTCB
Maintainability / Analyzability 3. Demands little effort to locate causes of failure in software	ISO/IEC 25023 - Audit record of error causes metric: NRCE = number of records of error causes in system operations NPRCE = number of records of error causes planned sufficiently to monitor the system status during its operation MANUT4 = NRCE / NPRCE ISO/IEC 25023 - Diagnostic function sufficiency metric: NFDF = number of functions for diagnosis of failures available for the system NPFDF = number of functions for diagnosis of failures predicted in system requirements MANUT5 = NFDF / NPFDF
Maintainability / Testability 4. The system can be efficiently tested after a modification	ISO/IEC 25023 - Test case coverage metric: NCT = number of test cases prepared to test system functionalities NTCT = total number of test cases estimated for functional verification of system Note: Include test cases for integration with other systems/applications MANUT6 = NCT / NTCT
Portability/Adaptability 1. The system operates in standard market environments (operational system, database, development tools, etc.) 2. The system presents independence and mobility to store and recover information (notebooks, tablets, PDAs, etc.)	ISO/IEC 25023 - Software environment adaptability metric: NFAS = number of system functions tested successfully in other software environments (SGBDs, operational systems, development environments, etc.), besides the native environment NTF = total number of functions in system. PORT1=NFAS/NTF ISO/IEC 25023 - Hardware environment adaptability metric: NFAH = number of functions successfully tested in the system in other hardware environments besides the native environment. NTF = total number of functions in system. PORT2=NFAH/NTF
Portability/ Capacity to be installed 3. Installation of system in user environment is easy and fast	ISO/IEC 25023 - Ease of installation metric: NIS = number of successful installations, where installation occurred according to user convenience and in adequate timeframe. NTIS = total number of installations and attempts to install system PORT3 = NIS / NTIS

The second column of Chart 1 describes the metric specifications for the indicators assessed using inspection procedures. 37 procedures are based on the use of metrics from the document ISO/IEC 25023(ISO, 2011b), four are based on rule ISO/IEC 9126-3 (ISO, 2003) and one was proposed by the author. Every metric is labeled with an abbreviation of the characteristic it refers to and numbered sequentially.

4.2. Feasibility study of inspection procedures used in assessment model

In this phase, a survey was carried out with information technology professionals from Hospital das Clínicas in the Medical School of Ribeirão Preto (HCFMRP-USP), to investigate the applicability of inspection procedures specified for the proposed assessment model, according to that specified in the method section.

The survey was performed at the Information and Analysis Center (CIA - Centro de Informações e Análises), the department responsible for information technology management at the hospital. The CIA provides the hospital with its own IT development and infrastructure that was gradually organized as from 1995, when the applications maintained by PRODESP were migrated. PRODESP previously offered information technology support for the hospital.

Currently, the CIA maintains 65 information systems in deployment or operation, 55 developed internally and 10 contracted from third-parties. As collaborators, it has 20 business and system analysts, 2 software quality engineers, 3 network administrators and 1 project manager, as well as a group of information technicians to provide support and service to the users.

The applications are developed in the Microsoft.Netplatform and the Oracle database manager, they use a UML (Unified Modelling Language) for analysis specifications and prototype techniques and tools for survey and specification of requirements. Some legacy Delphi systems are still maintained gradually being submitted to reengineering for technological update and review of functionalities.

The Senior Management of HCFMRP has sponsored policies to promote improvement of the software development process quality, with SBIS/CFM (SBIS, 2013) and MPS-BR (SOFTEX, 2013) certifications, to improve IT infrastructure development enhancement.

For the survey on applicability of inspection procedures proposed, three information systems were investigated. They support clinical and hospital processes at HCFMRP:

• Laboratory Information System (LIS - Sistema de Informações Laboratoriais): offers support for management of scheduled examinations and performed in the different clinical analysis laboratories at the hospital.

• Clinical Service Elaboration (EAC -Elaboração de Atendimento Clínico): shows the observations and evolutions in patient care pointed out by doctors, paramedics and nurses, through electronic forms.

• Surgical Procedure Management (CIRÚRGICO-3 - Gerenciamento de Procedimentos Cirúrgicos): for surgeries performed at the hospital, this system manages waiting lists, scheduling, notes on surgical procedures, issuing of surgical records, consumption of material as well as analytical and managerial reports.

Chart 2 describes the outcomes obtained in the survey performed using questionnaires, where the system analysts responsible for the applications were interviewed, according to that described in the method section. The chart enumerates the metrics for each quality attribute. Note that the same names attributed to the metrics in Chart 1 were used. The affirmative answers are indicated with the symbol "▲", while the negative ones with "▼". The symbol "▬" indicates that the metric does not apply to the system context and "?"indicates that the question was not answered by the interviewee. The last column summarizes the explanations from the respondents about compliance with procedures.

Chart 2
Compliance of assessment procedures with applications analyzed

Inspection procedures		Application			Observations/Comments from professionals interviewed
		LIS	EAC	CIR
Quality Attributes	Metrics
Functional supportability / Functional completeness	SUPFUNC1	▼	▲	▲	LIS and CIR3 have no requirement documentation or are not updated, different from EAC.
	SUPFUNC2	▼	▲	▼
	SUPFUNC3	▬	▲	▼
	SUPFUNC4	▼	▲	▲
	SUPFUNC5	▼	▲	▼
Functional supportability / Functional correctness	SUPFUNC6	▲	▲	▲	Feasible metric but difficult deployment.
Functional supportability /Functional adequacy	SUPFUNC7	▲	▬	▼	Depends on the information in requirement documents.
Performance efficiency/ Behavior with regard to time	EFDES1	▬	▼	▼	For the LIS analyst, performance is not a critical problem of the system; The analysts of EAC and CIR3 observe that there are no performance specifications in the non-functional requirements for these systems.
	EFDES2	▬	▼	▼
	EFDES3	▼	▼	▼
Performance efficiency/ Capacity	EFDES4	▲	▼	▼
Compatibility/ Coexistence	COMPAT1	▲	▲	▲	There was a consensus for these attributes.
Compatibility/ Interoperability	COMPAT2	▲	▲	▲
Usability/ Learnability	USAB1	▲	▲	▲	There was consensus in the feasibility of USAB1, USAB2, USAB3, USAB7 and USAB10 and in the unfeasibility of USAB8. For USAB4, there were questions about the customization concept. USAB5 and USAB6 depend on information found in requirement documentation, not always available.
Usability / Operability	USAB2	▲	▲	▲
	USAB3	▲	▲	▲
	USAB4	▼	?	▼
Usability / Protection from user error	USAB5	▼	▲	▼
	USAB6	▼	▲	▼
	USAB7	▲	▲	▲
	USAB8	▼	▼	▼
Usability / Esthetics in interface with user	USAB9	▼	▲	▼
Usability/ Accessibility	USAB10	▲	▲	▲
Reliability / Maturity	CONF1	?	▲	▲	For this characteristic, there was a consensus in the applicability of metrics for all attributes, except CONF5, which requires specifications in non-functional environment requirements.
	CONF2	?	▲	▲
	CONF3	▲	▲	▲
Reliability / Availability	CONF4	▲	▲	▲	For the metric CONF2 (MTBF), it was observed that the size of the system has to be considered, for its effectiveness.
Reliability / Tolerance to failure	CONF5	▼	▼	▼
Reliability / Recoverability	CONF6	▲	▲	▲
Reliability/ no questioning	SEG1	▲	▲	▬	There was a consensus for these attributes.
Safety / Confidentiality+ Authentication+Integrity	SEG1	?	▲	▲
	SEG2	?	▲	▲
Safety / accountability	SEG3	▲	▲	▲
Maintainability / Modularity + Modifiability	MANUT1	▲	▼	▼	There was a consensus only for MAINT3. For MAINT1 and MANUT2, it was questioned whether the metrics are in fact associable to the indicator. For MAINT4 and MAINT5, the absence of answers was due to lack of understanding/clarity in the definition of the metrics. Metric MAINT6, that assesses testability, depends on the availability of test cases planned for the target system.
	MANUT2	▲	▼	▼
Maintainability /Reusability	MANUT3	?	▲	▲
Maintainability/ Analyzability	MAINT4	?	▼	?
	MAINT5	?	▼	?
Maintainability / Testability	MAiNT6	▲	▼	▼
Portability / Adaptability	PORT1	▲	▬	▬	There was a consensus for these attributes.
	PORT2	▲	▲	▲
Portability/ Capac. to be installed	PORT3	▲	▲	▲

In absolute numbers, 69 feasibility indications were computed for the metrics proposed against 40 indications of non-feasibility (there were also 7 comments about absence of context of use and 10 questions were not answered). Graph 1 shows the distribution of answers per quality characteristic.

Graph 1
Distribution of responses in the model's quality attributes

As shown in graph 1, there was a positive consensus for all the metrics of Compatibility, Safety and Portability characteristics (fully compliant) and for most of the Usability sub-characteristics (partially compliant). It was observed that the applicability of metrics for Functional Supportability were conditioned to the existence of functional requirement documentation for the applications - not always available for the applications, which resulted in partial compliance with the procedures, as well as with two Usability metrics.

For the Reliability metric group, there was also a positive consensus, with the exception of the metric that assesses component redundancy, as there was no specification on the non-functional requirements of the infrastructure. For the metric MTBF (Media Time Between Failures), it was correctly observed that the procedure does not consider the size of the system and that the number of application functions must consider the measurement for calculation.

The absence of non-functional requirement specifications for response time, authentication time and simultaneous accesses made it unfeasible to use all the Performance Efficiency metrics. For the Maintainability characteristic, there was little feasibility: the outcomes show that the Testability depends on the availability of text cases for each system and indicate that some metric specifications in these groups must be reviewed, for better clarity, as this group concentrated the largest number of non-answered questions.

As suggested by this outcome, the analysts interviewed agree that many of the specified procedures can be adopted for assessment of the systems that address HCFMRP, also that they are aligned with the specifications recommended by certification SBIS/CFM (SBIS, 2013), recently implanted in the IT development and infrastructure areas.

5. FINAL REMARKS

For an assessment model for healthcare information systems based on the quality dimension of the product in rule ISO/IEC 25010 and in quality indicators researched in literature, this article describes the specification of a set of software inspection procedures associated to the indicators proposed and assesses the feasibility of these procedures at HCFMRP, a public hospital with regional coverage.

The architecture of the assessment model has as a distinguishing factor the application of assessment instruments to stakeholders with conflicting interests: its architecture predicts questionnaires directed towards different user profiles of the system and inspection procedures, with collection of software metrics with the system maintainers, using documental analysis, tests and software simulation use. This work includes as a study object the second group of assessment instruments for the model, directed towards application maintainers.

The feasibility study developed for systems that address HCFMRP suggests that most of the specified procedures are applicable to the hospital context, particularly the metrics for the attributes Compatibility, Reliability, Safety, Portability and Usability. A longitudinal measurement strategy, with a view of historical assessment data can guide improvement processes and reengineering of systems.

Future work includes assessment together with other stakeholders not included in the assessment, other assessment studies of applications and the submission of the model and outcomes of assessments to specialist panels in healthcare information, organized in focal groups, for validation and adjustments of the framework developed.

When covering the main quality aspects of the healthcare information system domain, with a holistic range of indicators, questionnaires and software metrics, this work can contribute as another reference of studies that involve assessment processes of the technical quality of healthcare software and other areas of application, with adaptations.

Many managers have as a motto that everything that is managed must be measured - after all, measuring allows for quantification and, consequently, more effective management. When identifying a demand to discuss the quality of healthcare information systems, it is also expected that the outcome of this work can add content and/or provide subsidies for projects that deal with standardization of assessment plans and monitoring of system quality and in enhancement projects of these software assets.

• Published by/ Publicado por: TECSI FEA USP - 2016 All rights reserved

REFERENCES

• Abouzahr, C., &Boerma, T. (2005).Health information systems: the foundations of public health".Bulletin of the World Health Organization, 83, 578-583. Retrieved from http://www.who.int/bulletin/volumes/83/8/en/index.html
  » http://www.who.int/bulletin/volumes/83/8/en/index.html
• Albertin, A.L.,& Moura, R.M. (1995). Administração de informática e seus fatores críticos de sucesso no setor bancário privado nacional.Revista de Administração de Empresas, 35(5), 22-29.
• Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., &König, J. (2003). Evaluation of health information systems-problems and challenges. International journal of medical informatics, 71(2), 125-135.
• Ammenwerth, E., Brender, J., Nykänen, P., Prokosch, H. U., Rigby, M., &Talmon, J. (2004). Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. International journal of medical informatics, 73(6), 479-491.
• Anderson J.,&Aidyn, C. (2005).Evaluating the organizational impact of healthcare information systemsNew York:Springer.
• Caldas, M. P., &Wood, T. J. (2000). Fads and fashions in management: the case of ERP. Revista de Administração de Empresas, 40(3), 8-17.
• Carvalho, A. O., &Eduardo, M. B. P. (1998).Sistemas de Informação em Saúde para Municípios. Série Saúde & Cidadania. v. 6São Paulo:Faculdade de Saúde Pública da Universidade de São Paulo.
• D'Souza, S. C., &Sequeira, A. H. (2011).Information Systems and Quality Management in Healthcare Organization: An Empirical Study. Journal of Technology Management for Growing Economies, 2(1), 47-60.
• Dallavalle, S. I. (2000).Regras de Negócio, um fator chave de sucesso no processo de desenvolvimento de sistemas de informaçãoAnais do XX ENEGEP - Encontro Nacional de Engenharia de Produção. São Paulo.
• Fichman, R. G.,Kohli, R., & Krishnan, R. (2011). The role of information systems in healthcare: current research and future trends.Information Systems Research, 22(3), 419-428.
• Hillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R., & Taylor, R. (2005). Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs, 24(5), 1103-1117.
• Hübner-Bloder, G., &Ammenwerth, E. (2009). Key Performance Indicators to Benchmark Hospital Information Systems-A Delphi Study. Methods of Information in Medicine, 48(6), 508-518.
• ISO (2011a).ISO/IEC 25010. Systems and software engineering -systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models Geneva:International Organization for Standardization.
• ISO (2003). ISO/IEC TR 9126-3. Software engineering - Product Quality - part 3: Internal metricsGeneva:International Organization for Standardization.
• ISO (2011b). ISO/IEC 25023. Systems and software engineering -systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of system and software product quality (in development) International Organization for Standardization.
• Lima, C. R. A., Schramm, J. M. A., Coeli, C. M., & Silva, M. E. M. (2009). Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cadernos de saúde pública, 25(10), 2095-2109.
• Malhotra, N. K. (2006). Pesquisa em Marketing: uma orientação aplicada Porto Alegre:Bookman.
• Morais, R. M., & Costa, A.L. (2013). Avaliação orientada a múltiplos stakeholders: um modelo para os sistemas de informação do Sistema Único de SaúdeIn Anais do XXXVII Encontro da ANPAD - EnANPAD 2013. Rio de Janeiro.
• Newell, S. (2011). Special section on healthcare information systems. The Journal of Strategic Information Systems, 20(2), 158-160.
• Oliveira, S. V. W. B., Arroyo, C. S., Oliveira, M. M. B., & Ferreira, C. H. (2011). Use and Development of Health Information Systems: The Experience of an Organizational Unit Responsible for the Technological Services atPublic Hospitals. Journal of Information Systems and Technology Management, 8(1) , 155-178.
• OMS (2013). Global Health Observatory Data Repository. Retrieved fromhttp://apps.who.int/ghodata/?vid=1901
  » http://apps.who.int/ghodata/?vid=1901
• OPAS (2011). A atenção à saúde coordenada pela APS: construindo as redes de atenção no SUSBrasília: Organização Pan-Americana da Saúde.
• Otieno, G. O., Hinako, T., Motohiro, A., Daisuke, K., &Keiko, N. (2008). Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. International journal of medical informatics, 77(10), 657-669.
• Pai, F. Y., & Huang, K. I. (2011). Applying the Technology Acceptance Model to the introduction of healthcare information systems. Technological Forecasting and Social Change, 78(4), 650-660.
• Ribière, V., LaSalle, A. J., Khorramshahgol, R., & Gousty, Y. (1999). Hospital Information Systems Quality: A customer satisfaction assessment tool. In Proceedings of the 32nd Annual Hawaii International Conference on System Sciences, 1999. HICSS32 (pp. 7-pp). IEEE.
• SBIS (2013). Manual de Certificação para Sistemas de Registro Eletrônico em Saúde (S-RES) Retrieved from http://www.sbis.org.br/certificacao/Manual_Certificacao_SBIS-CFM_2009_v3-3.pdf
  » http://www.sbis.org.br/certificacao/Manual_Certificacao_SBIS-CFM_2009_v3-3.pdf
• SOFTEX (2013). MPS.BR - Melhoria de Processo do Software Brasileiro. Guia GeralRetrieved from http://www.softex.br/mpsbr/_guias/guias/MPS.BR_Guia_Geral_2009.pdf
  » http://www.softex.br/mpsbr/_guias/guias/MPS.BR_Guia_Geral_2009.pdf
• Vergara, S. C. (2005). Projetos e Relatórios de Pesquisa em AdministraçãoSão Paulo:Atlas.
• Viitanen, J., Hyppönen, H., Lääveri, T., Vänskä, J., Reponen, J., &Winblad, I. (2011). National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. International journal of medical informatics, 80(10), 708-725.
• Wager, K. A., Lee, F. W., & Glaser, J. P. (2009). Health care information systems: a practical approach for health care management Wiley. com.
• Yusof, M. M., Papazafeiropoulou, A., Paul, R. J., &Stergioulas, L. K. (2008). Investigating evaluation frameworks for health information systems. international journal of medical informatics, 77(6), 377-385.

Publication Dates

• Publication in this collection
  Sep-Dec 2016

History

• Received
  03 Nov 2013
• Accepted
  26 Apr 2016