SciELO - Scientific Electronic Library Online

 
vol.49 special issueAtypical secretion of cortisol in Nursing professionalsOrganizational culture and climate for patient safety in Intensive Care Units author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

Share


Revista da Escola de Enfermagem da USP

Print version ISSN 0080-6234On-line version ISSN 1980-220X

Rev. esc. enferm. USP vol.49 no.spe São Paulo Dec. 2015

http://dx.doi.org/10.1590/S0080-623420150000700017 

Original Article

Interrater reliability of Nursing Activities Score among Intensive Care Unit health professionals

Confiabilidade inter observadores do Nursing Activities Score entre profissionais de saúde da Unidade de Terapia Intensiva

Confiabilidad inter-observador del Nursing Activities Score entre profesionales de la salud en una Unidad de Cuidado Intensivo

Marit Stuedahl1  2 

Sidsel Vold1  2 

Pål Klepstad1  2 

Siv Karlsson Stafseth3  4 

1St. Olav’s University Hospital, Department of Anesthesiology and Intensive Care Medicine, Trondheim, Norway.

2Norwegian University of Science and Technology, Faculty of Medicine, Department of Circulation and Medical Imaging, Trondheim, Norway.

3Oslo University Hospital, Division of Emergencies and Critical Care, Department of Research and Development, Oslo, Norway.

4University of Oslo, Faculty of Medicine, Institute of Clinical Medicine, Oslo, Norway.

ABSTRACT

Objective

To analyze the interrater reliability of NAS among critical care nurses and managers in an ICU.

Method

This was a methodological study performed in an adult, general ICU in Norway. In a random selection of patients, the NAS was scored on 101 patients by three raters: a critical care nurse, an ICU physician and a nurse manager. Interrater reliability was analyzed by agreement between groups and kappa statistics.

Results

The mean NAS were 88.4 (SD=16.2) and 88.7 (SD=24.5) respectively for the critical care nurses and nurse managers. A lower mean of 83.7 (SD=21.1) was found for physicians. The 18 medical interventions showed higher agreement between critical care nurses and physicians (85.6%), than between critical care nurses and nurse managers (78.7). In the five nursing activities the Kappa-coefficients were low for all activities in all compared groups.

Conclusion

The study indicated a satisfactory agreement of nursing workload between critical care nurses and managers.

Key words: Workload; Nursing, Team; Nursing Care; Intensive Care Units; Reproducibility of Results

RESUMO

Objetivo

Analisar a confiabilidade interobservadores do NAS entre enfermeiros intensivistas e administradores em UTI.

Método

Estudo metodológico desenvolvido em UTI geral, de adultos, na Noruega. Em uma amostra selecionada, o NAS foi aplicado em 101 pacientes por três avaliadores: enfermeira assistencial, médico intensivista e enfermeira gestora. A confiabilidade interobservadores foi analisada por meio do teste Kappa.

Resultados

As médias NAS foram 88,4(SD=16,2) e 88,7(SD=24,5) respectivamente para enfermeiros assistenciais e gestores. Os médicos obtiveram média NAS inferior (83,7; SD=21,1). As 18 intervenções médicas tiveram maior concordância entre enfermeiros assistenciais e médicos (85,6), comparativamente aos enfermeiros assistenciais e gestores (78,7). Nas cinco atividades de enfermagem, os coeficientes Kappa foram baixos em todas as atividades e entre todos os grupos.

Conclusão

O estudo mostrou confiabilidade interobservadores satisfatória para o NAS entre enfermeiros assistenciais e gestores.

Palavras-Chave: Carga de Trabalho; Equipe de Enfermagem; Cuidados de Enfermagem; Unidades de Terapia Intensiva; Reprodutibilidade dos Testes

RESUMEN

Objetivo

: Analizar la confiabilidad inter-observador del NAS entre administradores y enfermeras clínicas en la UCI.

Método

: Estudio metodológico desarrollado en una UCI general en Noruega. En una muestra seleccionada el NAS fue aplicado en 101 pacientes críticos por tres clases de evaluadores: Enfermeras asistenciales, médicos intensivistas y enfermeras gestoras. La confiabilidad inter-observador fue analizada mediante el test de Kappa.

Resultados

: Los promedios del NAS fueron 88,4(DE=16,2) y 88,7(DE=24,5) respectivamente para las enfermeras asistenciales y gestoras. Los médicos obtuvieron un promedio NAS inferior 83,7;DE=21,1). Las 18 intervenciones médicas tuvierón mayor concordancia entre las enfermeras asistenciales y los médicos (85,6), en comparación con las enfermeras asistenciales y gestoras (78,7). En las cinco actividades de enfermería, los coeficientes Kappa fueron bajos entodas las actividades y entre todos los grupos.

Conclusión

: El estudio mostró confiabilidad inter-observador satisfactorias para el NAS entre las enfermeras asistenciales y gestoras.

Palabras-clave: Carga de Trabajo; Grupo de Enfermería; Atención de Enfermería; Unidades de Cuidados Intensivos; Reproducibilidad de Resultados

INTRODUCTION

Health care is in continuous development and should meet increasing demands for changes in terms of technology, new treatments, and quality of care(1-2). Intensive care units (ICUs) are not exempt from such change requirements and their associated costs(3). The proportion of elderly ICU patients has increased and several of these suffer from complicating additional diseases. This might result in higher needs for ICU care(2). The major proportion of ICU costs is from staffing, especially nursing staffing(3). ICUs have different numbers of nurses based on patient case mix and national standards, where Norway and the United Kingdom represent the highest nurse staffing(3). Both the quality of care and the health of nurses are negatively associated with nurses’ workloads(4-5). A workday characterized by situations of stress as a result of understaffing, and therefore increased workloads for individual nurses, may increase the risk of unwanted medical incidents, and thus the quality of health services(6-9).

The actual workload for ICU nurses, in our study named ICU-CCN, relates to the nurse responsible for daily care of patients. ICU-CCN is therefore an important factor for ICU quality of care. It is of interest to know whether ICU managers precisely assess the ICU-CCNs workload(1,3). A widely used and validated instrument to assess ICU-CCN workload is the Nursing Activities Score (NAS)(10-14). Assessed at the individual patient level, the NAS measures the average time spent by nurses in performing activities related to therapeutic procedures; medical interventions and nursing activities. Medical interventions include oxygen-treatment, administration of medications, dialysis, and procedures such as lung-physiotherapy, care of drains or hemodynamic measurements. Nursing activities include monitoring, hygiene, mobilization, administrative activities, and psychological support for the patient and family. The NAS was validated in a multi center study in which 99 ICUs from 15 countries participated(12).

To our knowledge, there are no published studies that compare ICU managers’ and bedside nurses’ assessments of nursing workload using the NAS. Our hypothesis is that we will have a high agreement between health professionals on nurse workloads in the ICU. For this reason, the aim of this study was to analyze the reliability of NAS between ICU physicians (ICU-PHYs), ICU nurse managers (ICU-NMs), and ICU-CCNs in ICU patients.

METHOD

This methodological study with a prospective NAS data collection was exploratory in interrater reliability between groups of health professionals from an ICU. It was conducted at the general ICU at St. Olav’s University Hospital in Trondheim, Norway, a nine-bed adult ICU, which treats all patients’ categories except cardiothoracic surgery and organ transplant patients. The unit consists of single patient rooms. As is typical for a Norwegian ICU, the employees at this ICU are restricted to nurses and physicians with a nurse-patient ratio of 1.3:1. This ICU is a closed unit where ICU physicians are responsible for the medical treatment and care of patients. Patients eligible for inclusion in the study were those with a minimum length of stay (LOS) of 24 hours in the ICU, related to that NAS was scored per 24 hours. The patients were 18 years or older. For each study day, all eligible patients in the unit were identified and, of those, 3-6 patients were randomly selected to participate in the study. Each patient’s score was collected for one day and the very same patient could be randomized for another day with a new one day score. Data collection was carried out during a 6-week period starting September 2013 and included 101 patients. Patient characteristics regarding age, gender, and Simplified Acute Physiology Score (SAPS II)(15) were obtained from the medical records. SAPS II is a severity of illness system and stratifies patients within the first 24 hours following admission to the ICU. A higher score can indicate an increased severity and risk of mortality, resulting in a score between 0-163, and a predicted mortality between 0-100%.

The Nursing Activities Score (NAS)(12,14) was chosen for this study in the assessment of nursing workload. The NAS consists of 23 items with sub-items, each one scored from 1.3% to 32%, representing the percentage of time that one critical care nurse spends on a specified activity. The percentages obtained in all items are then added to obtain the final result, which ranges from 18.3 to 177%. The nursing workload related to medical interventions (18 items) in the NAS have only two categories: present or not present. Nursing activities (5 items) have two or three levels based on time consumed on nursing or the number of nurses who perform the activity. A Norwegian version of NAS was used for this study. That version was translated into Norwegian earlier in 2008 by a method for translation and cultural adaption(16). The chart was identical for all respondents(14).

All health professionals in the three groups were trained in scoring the NAS before the study, and their ability to score was tested in simulated cases. This study included 120 ICU-CCNs, 4 ICU-NMs, and 2 ICU-PHYs. All included patients were individually scored by health care professionals from each group and were “randomized” in the meaning of care and treatment of the selected study patients. ICU-CCNs on duty scored NAS on their patients at 9 pm, 6 am, and 2 pm. These three scores were summed up according to guidelines, and constituted the patients NAS for that 24 hour period. ICU-PHY and ICU-NM scored NAS at 2 pm, based on perceived nursing workload on the 3-6 randomly selected patients.

1) ICU-CCN. The nurse responsible for the daily care of the patient. Among the 120 nurses who participated, 66 were critical care nurses with a post-graduate education of 1.5 to 2 years, all called ICU-CCNs in the present study.

2) ICU-NM. The nurse manager who allocates nursing resources and coordinates patient transfers to and from other hospital units. The ICU has four ICU-NMs who are scheduled on a weekly rotation.

3) ICU-PHY. The physician with the overall medical responsibility in the ICU and who decides the medical care, allocates services from physicians, and coordinates medical care with the referring units. Two ICU-PHYs participated in the study, both anesthesiologists and intensive care medicine specialists.

PASW-Predictive Analytics Software statistics for Mac, version 21 was used for statistical analysis. Descriptive statistics were used to calculate frequencies and means. For inference analysis for differences in scores between the three groups of health professionals; for normal distributions the independent two-sample t-tests were performed and for skewed data one-way ANOVA with post-hoc Bonferroni (significant at <0.05). Interrater reliability was calculated using Kappa statistics and proportion of agreement(17). The Kappa-coefficient shows the agreement between assessments performed by the ICU-CCNs and the ICU-PHYs, and the ICU-CCNs and ICU-NMs. Kappa rates agreement ranged from 0 or less (no agreement) to 1 (perfect agreement)(18). In order to identify potential areas with high or low reliability, the different items were analyzed separately using Kappa statistics, and when not possible to calculate Kappa, proportion of agreement was used.

The Regional Committee for Research Ethics in Norway (No 2013/894) approved the study and waived the need for consent from the patients. Informed consent was obtained from all health professionals who participated. The study involved no intervention that affected the care for the patients. All charts and data were stored safely, anonymously, and we followed the regulations of protection from the hospital and law.

RESULTS

In the sample of 101 patients, 83.2% were male, the mean age was 56.6 years, and the mean SAPS II(15) was 42.2. The most frequent causes for admittance to the ICU were diagnoses related to respiratory and circulatory failure.

Table 1 shows that the mean NAS for total score showed no differences between ICU-CCNs and ICU-NMs (88.4 versus 88.7). ICU-PHYs scored significantly lower with a mean score of 83.7. For nursing activities (Table 1), ICU-CCNs and ICU-PHYs had similar scores (56.7 versus 54.3), while ICU-NMs scored a significant higher NAS (65.9). As shown in Table 1, the results of the medical intervention scores were similar between ICU-CCNs and ICU-PHYs (28.2 versus 26.4), while the ICU-NMs had a lower NAS (22.4).

Table 1 Descriptive statistics of NAS Total scores, Nursing activities, and Medical interventions in groups of health professionals - St. Olav University Hospital, Trondheim, Norway, 2013. 

NAS total score (%)
Health Professionals Registrations (N) Mean (SD) Median Min-Max
ICU-CCN 101 88.4 (16.2) 85.6 58.3-148.7
ICU-NM 101 88.7 (24.5) 80.5 54.3-145.0
ICU-PHY 101 83.7 (21.1)* 89.4 28.8-140.6
NAS Nursing activities (%)
Health Professionals Registrations (N) Mean (SD) Median Min-Max
ICU-CCN 101 56.7 (15.9) 48.9 33.4-110.5
ICU-NM 101 65.9 (24.7)* 59.3 32.8-118.6
ICU-PHY 101 54.3 (18.0) 57.3 12.8-103.7
NAS Medical interventions (%)
Health Professionals Registrations (N) Mean (SD) Median Min-Max
ICU-CCN 101 28.2 (4.4) 28.0 19.0-39.3
ICU-NM 101 22.4 (4.9) * 23.0 11.50-38.0
ICU-PHY 101 26.4 (5.3) 27.0 14.0-37.9

*p-value<0.05 for that group’s score compared to the other two groups.

The results related to interrater reliability for NAS nursing activities (e.g. five items) are presented in Table 2. The results indicate a higher proportion of agreement between ICU-CCNs and ICU-PHYs for the activities ‘hygiene’ and ‘support & care of patient & relatives’; and a higher proportion of agreement for activities related to mobilization and administrative tasks between ICU-CCNs and ICU-NMs. ICU-NM and ICU-PHY obtained a lower proportion of agreement on all activities, with best agreement on administrative tasks (item 8). Kappa-coefficients are low for all activities in all compared groups.

Table 2 Interrater Reliability of NAS Nursing Activities items among ICU´s health professionals - St. Olav University Hospital, Trondheim, Norway, 2013. 

Inter-rater reliability - NAS nursing activities items CCN/PHY agreement (%) Kappa CCN/NM agreement (%) Kappa NM/PHY agreement % Kappa
1 Monitoring and titration 100 ** 100 ** 100 **
4 Hygiene procedures 54.5 0.06* 32.7 -0.49* 39.6 0.05*
6 Mobilization and positioning 33.7 0.03* 49.5 0.16 39.6 0.02*
7 Support and care of patient and relatives 48.6 0.19 40.6 0.07* 41.6 0.12*
8 Administrative tasks 47.6 -0.36* 52.5 0.01* 49.5 0.02
Mean % agreement-nursing activities 56.9 - 55.1 - 54.1 -

* not significant p>0,05, ** Kappa not calculated because assumptions for calculating k are missing.

Table 3 shows interrater reliability for items related to medical interventions (e.g. 18 items). The proportion of agreement ranges from 65.3% to 100% for ICU-CCNs and ICU-PHYs, and from 25.8% to 100% for ICU-CCNs and ICU-NMs. ICU-CCNs and ICU-PHYs have higher proportions of agreement and higher kappa values than other compared groups. All groups show best Kappa values on medical intervention hemofiltration (Item 16).

Table 3 Interrater reliability of NAS Medical Interventions items among ICU health professionals - St. Olav University Hospital, Trondheim, Norway, 2013. 

Inter-rater reliability NAS - medical interventions items CCN/ PHY agreement (%) Kappa CCN/ NM agreement (%) Kappa NM/ PHY agreement (%) Kappa
2 Laboratory 68.3 0.24 25.8 -0.03* 31.7 0.17
3 Medications 97.0 -0.01* 69.3 0.04* 68.3 0.02*
5 Care of drains 74.3 0.34 74.3 0.32 70.3 0.24
9 Oxygen 100.0 ** 99.0 ** 100.0 **
10 Care of artificial airways 65.3 0.29 65.4 0.23 76.3 0.38
11 Treatment for improving lung function 86.1 0.48 78.3 0.19 80.2 0.32
12 Vasoactive medications IV 65.3 0.345 61.4 0.30 66.3 0.28
13 IV replacement of large fluid losses 94.1 -0.17* 93.1 -0.29* 99.0 0.66
14 Left atrium monitoring 99.0 ** 100.0 ** 99.0 **
15 CPR in the past 24h 100.0 ** 100.0 ** 100.0 **
16 Hemofiltration 98.0 0.90 96.0 0.80 94.0 0.69
17 Urine output measurement 95.0 ** 96.0 ** 93.1 0.19*
18 ICP-measurement 96.0 0.77 85.1 0.41 83.1 0.33
19 Treatment of acidosis/alkalosis 86.2 0.23 86.1 0.08* 92.1 0.17*
20 Intravenous hyper alimentation 75.3 0.50 65.4 0.30 58.4 0.18*
21 Enteral feeding 82.1 0.54 60.4 0.24 64.3 0.30
22 Specific interventions in ICU 72.3 0.21 71.3 0.15 77.2 **
23 Specific interventions outside ICU 87.1 0.47 90.1 0.49 85.2 0.32
Mean % agreement 18 items 85.63 78.72 79.91

* not significant p>0.05, ** Kappa not calculated because assumptions for calculating k are missing, ICP=Intra Cranial Pressure.

DISCUSSION

In this study about the interrater-reliability of NAS between ICU health professionals, we observed that ICU-CCNs and ICU-NMs had similar total scores (88.4% versus 88.7%), while ICU-PHYs scored lower NAS (83.7%). The 18 medical intervention results showed a better agreement between ICU-CCNs and ICU-PHYs compared to that obtained between ICU-CCNs and ICU-NMs. The opposite association was present for the nursing activities: mobilization and administrative tasks. ICU-PHYs and ICU-NMs both underestimated the care of families and relatives.

Nursing workload is difficult to assess, and several patient classification systems have been used(3,19-20). To explore different perspectives, e.g. ICU managers and ICU bedside nurses, this study used the NAS for assessment of workload. NAS is recommended for assessment of workload from other studies in Norway(14) and internationally(6,19,21).

In this study, the ICU-PHYs mean NAS total scores was significantly lower than for ICU-CCNs. This is the first study to publish NAS scored by ICU physicians. Previous studies involving physicians have used other scoring systems such as Nine Equivalents of Nursing Manpower Use Score (NEMS)(20) and SAPS II(15). Workload can be measured by NEMS but preferably for trends at a unit level and will only describe 35-40% of nursing time(12). NAS scores are assessed at patient level, so the weights represent time spent on workload for nursing activities, and the score represents 81% of workload for nursing to ICU patients(12). The ability of ICU physicians to know how much time nurses spend on different activities can be questioned. The physicians in ICUs are always present in the unit, but are in charge of critical patients in all rooms. Therefore, both because of a lack of professional insight about nursing and because of less detailed knowledge of each individual patient activity, the ICU physicians’ estimation of time spent on nursing activities may be limited. The results in this study are of interest to this discussion and might indicate that physicians tend to underestimate the total workload required for each ICU patient. However, the total mean score for ICU-PHYs is 4.7% lower than for ICU-CCNs, which is equivalent to about one hour per 24h for each patient(12). This difference may not have any important clinical significance.

The items related to NAS nursing activities had the lowest agreement between compared groups, and similar results were found in another study(11). The assessments of nursing activities, for example hygiene or support & care of patients & relatives, are likely to be influenced by the ICU-CCN’s individual judgment of needs, and might be difficult for ICU managers to assess. Another factor is that these nursing activities mainly take place behind closed doors, hence they may be harder to assess for ICU-PHYs and ICU-NMs who work outside the patient rooms. This may explain why the ICU-PHYs and ICU-NMs tend to underestimate resources used for mobilization & positioning, and support & care of patients & relatives as compared to ICU-CCNs, while they tend to overestimate resources used for hygiene and administrative tasks. This is important for the future organization of ICUs because support and care of patients and relatives (item 7) is a nursing activity that will become more demanding due to the use of less sedation for ICU patients, new methods for treatment such as non-invasive-ventilation that requires a high level of cooperation between ICU-CCNs and patients, and more access to the ICU room for relatives(22).

The physicians who are responsible for prescribing treatment and medication could easily identify medical interventions in the NAS. When it comes to the proportion of agreement and kappa-values, for medical interventions, the score was generally good between ICU-CCNs and ICU-PHYs, indicating close teamwork in the care of ICU patients. This teamwork is essential for reducing complications and mortality(2). As expected, the assessments of medical interventions by ICU-CCNs and ICU-NMs showed a lower proportion of agreement than between ICU-CCNs and ICU-PHYs. This could be explained because the medical interventions are often coordinated by the ICU physicians.

The results from the present study might have an impact on allocating resources of nurses. The cost of staffing, especially nurses(3) in ICUs is dependent on patient classification of workload. The ICU-NMs have a more administrative role, allocating nursing resources and coordinating with other units and hospitals, so one of the results for the score on administrative tasks (item 8) was in line with the expected one. In the unit in which the study was performed, the physical environment, with single rooms in a long hallway, can be difficult for the nurse in charge to have detailed information about all ongoing treatments administered inside those rooms. The best agreement among all the compared groups were easily identifiable medical interventions, like dialysis, and interventions that require a lot of nursing resources and assistance by physicians (interventions outside the ICU). These findings may suggest that the physical environment plays an important role when assessing the workload provided to ICU patients and allocating resources. The use of NAS to give hospital managers more accurate and detailed information about nursing workload is one of this study’s recommendations.

To use tools for estimating time in workload assessment can be both objective and subjective. Earlier studies on interrater-reliability and assessment tools(23) also indicate that time and workload assessments estimating time intervals have a lower reliability than assessments based on objective data. NAS can be critized because of the five items of nursing activities and becauase it can be time-consuming for nurses. Nonetheless, this research has followed the guidelines(12) and the Norwegian guideline(14).

This study has some limitations. The data were collected at a single ICU and NAS was recently introduced at the unit. On the other hand, the differences in each group of health professionals for knowledge, education, or experience in NAS registrations were not measured or analyzed.

CONCLUSION

The study indicates a satisfactory agreement about the assessment of nurses’ workload measured by NAS in a general adult ICU between critical care nurses and ICU managers. Further research focusing on reliability and validation tests in multicenter studies could confirm our results.

REFERENCES

1. Dahlgaard JJ, Pettersen J, Dahlgaard-Park SM. Quality and lean health care: A system for assessing and improving the health of healthcare organisations. Total Qual Manag Bus Excell. 2011;22(6):673-89. [ Links ]

2. Alameddine M, Dainty KN, Deber R, Sibbald WJ. The intensive care unit work environment: Current challenges and recommendations for the future. J Crit Care. 2009;24(2):243-8. [ Links ]

3. Reis Miranda D, Jegers M. Monitoring costs in the ICU: a search for a pertinent methodology. Acta Anaesthesiol Scand. 2012;56(9):1104-13. [ Links ]

4. Hoonakker P, Carayon P, Gurses AP, Brown R, Khunlertkit A, McGuire K, et al. Measuring workload of ICU nurses with a questionnaire survey: the NASA Task Load Index (TLX). IIE Trans Healthc Syst Eng. 2011;1(2):131-43. [ Links ]

5. Inoue KC, Versa GL, Matsuda LM. Stress level among intensive care nurses in the municipality of Parana (Brazil). Invest Educ Enferm. 2014;32(1):69-77. [ Links ]

6. Altafin JA, Grion CM, Tanita MT, Festti J, Cardoso LT, Veiga CF, et al. Nursing Activities Score and workload in the intensive care unit of a university hospital. Rev Bras Ter Intensiva. 2014;26(3):292-8. [ Links ]

7. Cremasco MF, Wenzel F, Zanei SS, Whitaker IY. Pressure ulcers in the intensive care unit: the relationship between nursing workload, illness severity and pressure ulcer risk. J Clin Nurs. 2013;22(15-16):2183-91. [ Links ]

8. Daud-Gallotti RM, Costa SF, Guimaraes T, Padilha KG, Inoue EN, Vasconcelos TN, et al. Nursing workload as a risk factor for healthcare associated infections in ICU: a prospective study. PloS One. 2012;7(12):e52342. [ Links ]

9. Silva MCM, Sousa RMCd, Padilha KG. Factors associated with death and readmission into the Intensive Care Unit. Rev Latino Am Enfermagem. 2011;19(4):911-9. [ Links ]

10. Debergh DP, Myny D, Van Herzeele I, Van Maele G, Reis Miranda D, Colardyn F. Measuring the nursing workload per shift in the ICU. Intensive Care Med. 2012;38(9):1438-44. [ Links ]

11. Ducci AJ, Padilha KG. Nursing activities score: a comparative study about retrospective and prospective applications in intensive care units. Acta Paul Enferm. 2008;21(4):581-7. [ Links ]

12. Miranda DR, Nap R, de Rijk A, Schaufeli W, Iapichino G. Nursing activities score. Crit Care Med. 2003;31(2):374-82. [ Links ]

13. Carmona-Monge FJ, Jara-Pérez A, Quirós-Herranz C, Rollán-Rodríguez G, Cerrillo-González I, García-Gómez S, et al. Assessment of nursing workload in three groups of patients in a Spanish ICU using the Nursing Activities Score Scale. Rev Esc Enferm USP. 2013;47(2):335-40. [ Links ]

14. Stafseth SK, Solms D, Bredal IS. The characterisation of workloads and nursing staff allocation in intensive care units: a descriptive study using the Nursing Activities Score for the first time in Norway. Intensive Crit Care Nurs. 2011;27(5):290-4. [ Links ]

15. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270(24):2957-63. [ Links ]

16. Fagerstrøm L, Vainikainen P. Nurses’ experiences of nonpatient factors that affect nursing workload: a study of the PAONCIL instrument’s nonpatient factors. Nurs Res Pract [Internet]. 2014 [cited 2015 Feb 22]:167764. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4090478/ [ Links ]

17. Polit DF, Beck CT. Nursing research: principles and methods. 10th ed. Philadelphia: Lippincott Williams &Wilkins; 2012. [ Links ]

18. Moher D, Altman D, Schulz K, Simera IAWE. Guidelines for reporting health research: a user’’s manual. Hoboken: Wiley; 2014. [ Links ]

19. Kakushi LE, Evora YD. Direct and indirect nursing care time in an intensive care unit. Rev Latino Am Enfermagem. 2014;22(1):150-7. [ Links ]

20. Reis Miranda D, Moreno R, Iapichino G. Nine equivalents of nursing manpower use score (NEMS). Intensive Care Med. 1997;23(7):760-5. [ Links ]

21. Nogueira LS, Domingues CA, Poggetti RS, Sousa RM. Nursing workload in intensive care unit trauma patients: analysis of associated factors. PloS One. 2014;9(11):e112125. [ Links ]

22. Salgado DR, Favory R, Goulart M, Brimioulle S, Vincent JL. Toward less sedation in the intensive care unit: a prospective observational study. J Crit Care. 2011;26(2):113-21. [ Links ]

23. Fischer M, Ruegg S, Czaplinski A, Strohmeier M, Lehmann A, Tschan F, et al. Inter-rater reliability of the Full Outline of UnResponsiveness score and the Glasgow Coma Scale in critically ill patients: a prospective observational study. Crit Care. 2010;14(2):R64. [ Links ]

Received: May 18, 2015; Accepted: August 21, 2015

Correspondence Addressed to: Siv K. Stafseth. Oslo University Hospital, Box 4950 Nydalen. 0424 - Oslo, Norway. sistaf@ous-hf.no

Creative Commons License This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.