Interrater reliability of Nursing Activities Score among Intensive Care Unit health professionals

Objective: To analyze the interrater reliability of NAS among critical care nurses and managers in an ICU. Method: This was a methodological study performed in an adult, general ICU in Norway. In a random selection of patients, the NAS was scored on 101 patients by three raters: a critical care nurse, an ICU physician and a nurse manager. Interrater reliability was analyzed by agreement between groups and kappa statistics. Results: The mean NAS were 88.4 (SD=16.2) and 88.7 (SD=24.5) respectively for the critical care nurses and nurse managers. A lower mean of 83.7 (SD=21.1) was found for physicians. The 18 medical interventions showed higher agreement between critical care nurses and physicians (85.6%), than between critical care nurses and nurse managers (78.7). In the five nursing activities the Kappa-coefficients were low for all activities in all compared groups. Conclusion: The study indicated a satisfactory agreement of nursing workload between critical care nurses and managers. DeScriPtOrS Workload; Nursing, Team; Nursing Care; Intensive Care Units; Reproducibility of Results. Correspondence Addressed to: Siv K. Stafseth Oslo University Hospital, Box 4950 Nydalen 0424 Oslo, Norway sistaf@ous-hf.no USP


INTRODUCTION
Health care is in continuous development and should meet increasing demands for changes in terms of technology, new treatments, and quality of care (1)(2) .Intensive care units (ICUs) are not exempt from such change requirements and their associated costs (3) .The proportion of elderly ICU patients has increased and several of these suffer from complicating additional diseases.This might result in higher needs for ICU care (2) .The major proportion of ICU costs is from staffing, especially nursing staffing (3) .ICUs have different numbers of nurses based on patient case mix and national standards, where Norway and the United Kingdom represent the highest nurse staffing (3) .Both the quality of care and the health of nurses are negatively associated with nurses' workloads (4)(5) .A workday characterized by situations of stress as a result of understaffing, and therefore increased workloads for individual nurses, may increase the risk of unwanted medical incidents, and thus the quality of health services (6)(7)(8)(9) .
The actual workload for ICU nurses, in our study named ICU-CCN, relates to the nurse responsible for daily care of patients.ICU-CCN is therefore an important factor for ICU quality of care.It is of interest to know whether ICU managers precisely assess the ICU-CCNs workload (1,3) .A widely used and validated instrument to assess ICU-CCN workload is the Nursing Activities Score (NAS) (10)(11)(12)(13)(14) .Assessed at the individual patient level, the NAS measures the average time spent by nurses in performing activities related to therapeutic procedures; medical interventions and nursing activities.Medical interventions include oxygen-treatment, administration of medications, dialysis, and procedures such as lung-physiotherapy, care of drains or hemodynamic measurements.Nursing activities include monitoring, hygiene, mobilization, administrative activities, and psychological support for the patient and family.The NAS was validated in a multi center study in which 99 ICUs from 15 countries participated (12) .
To our knowledge, there are no published studies that compare ICU managers' and bedside nurses' assessments of nursing workload using the NAS.Our hypothesis is that we will have a high agreement between health professionals on nurse workloads in the ICU.For this reason, the aim of this study was to analyze the reliability of NAS between ICU physicians (ICU-PHYs), ICU nurse managers (ICU-NMs), and ICU-CCNs in ICU patients.

METHOD
This methodological study with a prospective NAS data collection was exploratory in interrater reliability between groups of health professionals from an ICU.It was conducted at the general ICU at St. Olav's University Hospital in Trondheim, Norway, a nine-bed adult ICU, which treats all patients' categories except cardiothoracic surgery and organ transplant patients.The unit consists of single patient rooms.As is typical for a Norwegian ICU, the employees at this ICU are restricted to nurses and physicians with a nurse-patient ratio of 1.3:1.This ICU is a closed unit where ICU physicians are responsible for the medical treatment and care of patients.Patients eligible for inclusion in the study were those with a minimum length of stay (LOS) of 24 hours in the ICU, related to that NAS was scored per 24 hours.The patients were 18 years or older.For each study day, all eligible patients in the unit were identified and, of those, 3-6 patients were randomly selected to participate in the study.Each patient's score was collected for one day and the very same patient could be randomized for another day with a new one day score.Data collection was carried out during a 6-week period starting September 2013 and included 101 patients.Patient characteristics regarding age, gender, and Simplified Acute Physiology Score (SAPS II) (15) were obtained from the medical records.SAPS II is a severity of illness system and stratifies patients within the first 24 hours following admission to the ICU.A higher score can indicate an increased severity and risk of mortality, resulting in a score between 0-163, and a predicted mortality between 0-100%.
The Nursing Activities Score (NAS) (12,14) was chosen for this study in the assessment of nursing workload.The NAS consists of 23 items with sub-items, each one scored from 1.3% to 32%, representing the percentage of time that one critical care nurse spends on a specified activity.The percentages obtained in all items are then added to obtain the final result, which ranges from 18.3 to 177%.The nursing workload related to medical interventions (18 items) in the NAS have only two categories: present or not present.Nursing activities (5 items) have two or three levels based on time consumed on nursing or the number of nurses who perform the activity.A Norwegian version of NAS was used for this study.That version was translated into Norwegian earlier in 2008 by a method for translation and cultural adaption (16) .The chart was identical for all respondents (14) .
All health professionals in the three groups were trained in scoring the NAS before the study, and their ability to score was tested in simulated cases.This study included 120 ICU-CCNs, 4 ICU-NMs, and 2 ICU-PHYs.All included patients were individually scored by health care professionals from each group and were "randomized" in the meaning of care and treatment of the selected study patients.ICU-CCNs on duty scored NAS on their patients at 9 pm, 6 am, and 2 pm.These three scores were summed up according to guidelines, and constituted the patients NAS for that 24 hour period.ICU-PHY and ICU-NM scored NAS at 2 pm, based on perceived nursing workload on the 3-6 randomly selected patients.
1) ICU-CCN.The nurse responsible for the daily care of the patient.Among the 120 nurses who participated, 66 were critical care nurses with a post-graduate education of 1.5 to 2 years, all called ICU-CCNs in the present study.
2) ICU-NM.The nurse manager who allocates nursing resources and coordinates patient transfers to and from other hospital units.The ICU has four ICU-NMs who are scheduled on a weekly rotation.
3) ICU-PHY.The physician with the overall medical responsibility in the ICU and who decides the medical care, allocates services from physicians, and coordinates medical care with the referring units.Two ICU-PHYs participated in the study, both anesthesiologists and intensive care medicine specialists.
PASW-Predictive Analytics Software statistics for Mac, version 21 was used for statistical analysis.Descriptive statistics were used to calculate frequencies and means.For inference analysis for differences in scores between the three groups of health professionals; for normal distributions the independent two-sample t-tests were performed and for skewed data one-way ANOVA with post-hoc Bonferroni (significant at <0.05).Interrater reliability was calculated using Kappa statistics and proportion of agreement (17) .The Kappa-coefficient shows the agreement between assessments performed by the ICU-CCNs and the ICU-PHYs, and the ICU-CCNs and ICU-NMs.Kappa rates agreement ranged from 0 or less (no agreement) to 1 (perfect agreement) (18) .In order to identify potential areas with high or low reliability, the different items were analyzed separately using Kappa statistics, and when not possible to calculate Kappa, proportion of agreement was used.
The Regional Committee for Research Ethics in Norway (No 2013/894) approved the study and waived the need for consent from the patients.Informed consent was obtained from all health professionals who participated.The study involved no intervention that affected the care for the patients.All charts and data were stored safely, anonymously, and we followed the regulations of protection from the hospital and law.

RESULTS
In the sample of 101 patients, 83.2% were male, the mean age was 56.6 years, and the mean SAPS II (15) was 42.2.The most frequent causes for admittance to the ICU were diagnoses related to respiratory and circulatory failure.
Table 1 shows that the mean NAS for total score showed no differences between ICU-CCNs and ICU-NMs (88.4 versus 88.7).ICU-PHYs scored significantly lower with a mean score of 83.7.For nursing activities (Table 1), ICU-CCNs and ICU-PHYs had similar scores (56.7 versus 54.3), while ICU-NMs scored a significant higher NAS (65.9).As shown in Table 1, the results of the medical intervention scores were similar between ICU-CCNs and ICU-PHYs (28.2 versus 26.4), while the ICU-NMs had a lower NAS (22.4).
The results related to interrater reliability for NAS nursing activities (e.g.five items) are presented in Table 2.
The results indicate a higher proportion of agreement between ICU-CCNs and ICU-PHYs for the activities 'hygiene' and 'support & care of patient & relatives'; and a higher proportion of agreement for activities related to mobilization and administrative tasks between ICU-CCNs and ICU-NMs.ICU-NM and ICU-PHY obtained a lower proportion of agreement on all activities, with best agreement on administrative tasks (item 8).Kappa-coefficients are low for all activities in all compared groups.

DISCUSSION
In this study about the interrater-reliability of NAS between ICU health professionals, we observed that ICU-CCNs and ICU-NMs had similar total scores (88.4% versus 88.7%), while ICU-PHYs scored lower NAS (83.7%).The 18 medical intervention results showed a better agreement between ICU-CCNs and ICU-PHYs compared to that obtained between ICU-CCNs and ICU-NMs.The opposite association was present for the nursing activities: mobilization and administrative tasks.ICU-PHYs and ICU-NMs both underestimated the care of families and relatives.
Nursing workload is difficult to assess, and several patient classification systems have been used (3,(19)(20) .To explore different perspectives, e.g.ICU managers and ICU bedside nurses, this study used the NAS for assessment of workload.NAS is recommended for assessment of workload from other studies in Norway (14) and internationally (6,19,21) .
In this study, the ICU-PHYs mean NAS total scores was significantly lower than for ICU-CCNs.This is the first study to publish NAS scored by ICU physicians.Previous studies involving physicians have used other scoring systems such as Nine Equivalents of Nursing Manpower Use Score (NEMS) (20) and SAPS II (15) .Workload can be measured by NEMS but preferably for trends at a unit level and will only describe 35-40% of nursing time (12) .NAS scores are assessed at patient level, so the weights represent time spent on workload for nursing activities, and the score represents 81% of workload for nursing to ICU patients (12) .The ability of ICU physicians to know how much time nurses spend on different activities can be questioned.The physicians in ICUs are always present in the unit, but are in charge of critical patients in all rooms.Therefore, both because of a lack of professional insight about nursing and because of less detailed knowledge of each individual patient activity, the ICU physicians' estimation of time spent on nursing activities may be limited.The results in this study are of interest to this discussion and might indicate that physicians tend to underestimate the total workload required for each ICU patient.However, the total mean score for ICU-PHYs is 4.7% lower than for ICU-CCNs, which is equivalent to about one hour per 24h for each patient (12) .This difference may not have any important clinical significance.
The items related to NAS nursing activities had the lowest agreement between compared groups, and similar results were found in another study (11) .The assessments of nursing activities, for example hygiene or support & care of patients & relatives, are likely to be influenced by the ICU-CCN's individual judgment of needs, and might be difficult for ICU managers to assess.Another factor is that these nursing activities mainly take place behind closed doors, hence they may be harder to assess for ICU-PHYs and ICU-NMs who work outside the patient rooms.This may explain why the ICU-PHYs and ICU-NMs tend to underestimate resources used for mobilization & positioning, and support & care of patients & relatives as compared to ICU-CCNs, while they tend to overestimate resources used for hygiene and administrative tasks.This is important for the future organization of ICUs because support and care of patients and relatives (item 7) is a nursing activity that will become more demanding due to the use of less sedation for ICU patients, new methods for treatment such as non-invasive-ventilation that requires a high level of cooperation between ICU-CCNs and patients, and more access to the ICU room for relatives (22) .
The physicians who are responsible for prescribing treatment and medication could easily identify medical interventions in the NAS.When it comes to the proportion of agreement and kappa-values, for medical interventions, the score was generally good between ICU-CCNs and ICU-PHYs, indicating close teamwork in the care of ICU patients.This teamwork is essential for reducing complications and mortality (2) .As expected, the assessments of medical interventions by ICU-CCNs and ICU-NMs showed a lower proportion of agreement than between ICU-CCNs and ICU-PHYs.This could be explained because the medical interventions are often coordinated by the ICU physicians.
The results from the present study might have an impact on allocating resources of nurses.The cost of staffing, especially nurses (3) in ICUs is dependent on patient classification of workload.The ICU-NMs have a more administrative role, allocating nursing resources and coordinating with other units and hospitals, so one of the results for the score on administrative tasks (item 8) was in line with the expected one.In the unit in which the study was performed, the physical environment, with single rooms in a long hallway, can be difficult for the nurse in charge to have detailed information about all ongoing treatments administered inside those rooms.
The best agreement among all the compared groups were easily identifiable medical interventions, like dialysis, and interventions that require a lot of nursing resources and assistance by physicians (interventions outside the ICU).These findings may suggest that the physical environment plays an important role when assessing the workload provided to ICU patients and allocating resources.The use of NAS to give hospital managers more accurate and detailed information about nursing workload is one of this study's recommendations.
To use tools for estimating time in workload assessment can be both objective and subjective.Earlier studies on interrater-reliability and assessment tools (23) also indicate that time and workload assessments estimating time intervals have a lower reliability than assessments based on objective data.NAS can be critized because of the five items of nursing activities and becauase it can be time-consuming for nurses.Nonetheless, this research has followed the guidelines (12) and the Norwegian guideline (14) .
This study has some limitations.The data were collected at a single ICU and NAS was recently introduced at the unit.On the other hand, the differences in each group of health professionals for knowledge, education, or experience in NAS registrations were not measured or analyzed.

Table 1 -
Descriptive statistics of NAS Total scores, Nursing activities, and Medical interventions in groups of health professionals -St.Olav University Hospital, Trondheim, Norway, 2013.
*p-value<0.05 for that group's score compared to the other two groups.

Table 2 -
Interrater Reliability of NAS Nursing Activities items among ICU´s health professionals -St.Olav University Hospital, Trondheim, Norway, 2013.
* not significant p>0,05, ** Kappa not calculated because assumptions for calculating k are missing.

Table 3
shows interrater reliability for items related to medical interventions (e.g.18 items).The proportion of agreement ranges from 65.3% to 100% for ICU-CCNs and ICU-PHYs, and from 25.8% to 100% for ICU-CCNs and ICU-NMs.ICU-CCNs and ICU-PHYs have higher proportions of agreement and higher kappa values than other compared groups.All groups show best Kappa values on medical intervention hemofiltration (Item 16).

Table 3 -
Interrater reliability of NAS Medical Interventions items among ICU health professionals -St.Olav University Hospital, Trondheim, Norway, 2013.
* not significant p>0.05, ** Kappa not calculated because assumptions for calculating k are missing, ICP=Intra Cranial Pressure.