Reliability analysis of the Manchester Triage System: inter-observer and intra-observer agreement 1

ABSTRACT Objective: To analyze the reliability of the Manchester Triage System to determine the priority of patients in emergency services. Method: This is a reliability study with a sample of 361 nurses. The data were collected in three stages and the questionnaires were applied using the electronic software. The agreement was measured by the evaluation of clinical cases. The outcomes evaluated were agreement with the gold standard and intra-observer in the indication of the flowchart, discriminator, and level of risk. Data were submitted to univariate and bivariate analyses. The agreement was measured by the Kappa index. Results: The external and internal reliability of the protocol ranged from moderate to substantial (Kappa: 0.55-0.78). The time of professional experience as a nurse, in emergency services and in the classification of risk were associated with external and internal reliability. The correct choice of the discriminator influenced the correct indication of the risk level (R² = 0.77, p <0.0001) more than the correct choice of the flowchart (R² = 0.16, p <0.0001). Conclusion: The reliability of the Manchester Triage System ranged from moderate to substantial and it was influenced by the clinical experience of the nurse. The protocol is safe for defining clinical priorities using different classification flowcharts.

by five and assigns the target color and time for medical care at each level. It is structured in flowcharts with discriminators that guide the collection and analysis of information to define the clinical priority of the patient (4) .
In a context of demand for services that is greater than supply and with limited assistance resources, patient triage must happen accurately to ensure care according to the patient´s real need (5) . Thus, nurses´ decision-making in risk classification should be guided by a reliable support system capable of accurately measuring the patient´s degree of priority.
Reliability is considered the main criterion for testing the quality of measuring instruments. It is the instrument´s ability to measure consistently and accurately what it is intended to measure and to reproduce a result consistently in time and space, or with different observers (6) .
Although MTS was developed two decades ago and it is widely used in different countries as a nurse support system for decision-making in triage, its reliability has been little questioned (7) . An integrative review has indicated that the reliability of the MTS ranged from moderate to near-perfect, showing the need for further studies to identify changes required in the protocol and to increase safety in the management of patients´ clinical risk in emergency departments (8) .
There are variations in the studies on the reliability of the MTS that indicate a gap on the real reliability of this triage scale (7)(8) . Therefore, it is questioned: What is the reliability of the MTS to determine the degree of priority of the patient, considering the internal agreement and agreement between nurses who use this protocol? Thus, this study was designed to analyze the reliability of MTS to determine the priority of patients in emergency services. The sample was calculated considering 95% confidence interval, 5% margin of error, an estimate of correct answers between observers and a gold standard of 75% and an estimate of intra-observer scores of 85%. The estimate correct answer used was based on a previously performed reference study (9) . The reliability of the MTS was measured by the stability parameter, which consists of administering the same instrument to the same subjects under similar conditions on two or more occasions (10) . Clinical cases were evaluated by nurses at two different times (test and re-test).  instructions on how to proceed to answer the questions.

Method
Two data collection instruments titled "Clinical cases" and "Professional profile of nurses" were configured in the SurveyGizmo ® system.

Results
Most of the 361 nurses who participated in the study (294 -81.44%) were female, aged between 23 and 62 years old (mean: 34.16 years + 8.17 years). Table 1 shows the characterization of the nurses in the study, according to the variables of the professional profile investigated.   The linear regression analysis showed that there is a In the assessment of the external and internal reliability of the MTS, the overtriage was more frequent in level V of severity (blue color), occurring in 17% to 18% of the cases. Undertriage was more frequent in level II of severity (orange color), occurring in 27% of the cases. The patients were screened to a level above and one level below the actual severity of the case.
The external and internal reliability of the MTS was measured by calculating the kappa index for the choice of the flowchart, discriminator, and risk level (Table 3).   When assessing the external reliability of the MTS to choose the classification flowchart, nurses with five to ten years of experience in emergency services presented a greater agreement with the gold standard, when compared to those with less than one year of experience, but, for both groups, an agreement was substantial (Table 4).
Nurses with less than one year of professional experience presented a lower agreement with the gold standard when compared to the other groups when choosing the discriminator. Similarly, nurses with less than one year of experience in emergency services agreed less to the gold standard in discriminator selection than those with one to five and five to ten years of experience, and the higher the time of experience, the higher the kappa values found. There was no difference between those who had never acted and those who had acted for less than one year in the classification of risk and the agreement with the gold standard in the classification discriminator. However, those who work less than a year have been more compliant with the gold standard than those who have operated for more than ten years. Nurses with experience in the risk classification between five and ten years presented the highest values of kappa with the gold standard in the discriminator choice (Table 4).
The internal reliability of the MTS was evaluated by 153 (42.4%) of the 361 nurses who evaluated the external reliability of the MTS, and the agreement found ranged from moderate to substantial (Table 3).
Most nurses who participated in this stage of the study had between five years of professional experience as a nurse (43.8%), emergency services (58.2%), and risk classification (59.5%). Table 5 shows the variables in which significant associations between the Kappa values and the choice of the flowchart, discriminator, and level of risk were found.   (Table 5).

Discussion
Most of the nurses who participated in this study (79.23%) had between one and ten years of The triage scales or systems make up the decision support systems of nurses in the triage (2) . In this study, was found in a meta-analysis, in which the frequency of overtriage using MTS was 46.65 and undertriage was 12.86% (5) . MTS is useful in the triage of patients in emergency services, but the classification of patients to levels above or below the actual level still occurs, being more frequent the cases of overtriage (13) . Screening patients above the correct priority level can lead to unnecessary use of resources in emergency services (14) .
An underestimation of the level of risk may increase the risk of adverse consequences to patients, such as delayed care and the lack of adequate resources for their severity (15) .
The kappa values found to indicate that the external and internal reliability of the MTS ranged from moderate to substantial, with slightly higher values of kappa for intra-observer agreement (Table 3). A meta-analysis study conducted to assess the reliability of the MTS found that the overall agreement of the MTS was substantial (k=0.75). However, the authors emphasized that in the evaluated studies, the weighted kappa was used, which overestimate the reliability. Therefore, it is more prudent to consider that the current reliability of the MTS is moderate. In this study, the intra-observer agreement was also larger and near perfect, than the inter-observer agreement. The reliability among nurses was substantial (k=0.78), and among nurses and experts, it was almost perfect (k=0.86). The agreement was nearly perfect in the studies that measured it by the evaluation of patients in clinical practice (k=0.86) and it was substantial in the studies that evaluated agreement by the evaluation of clinical cases (k=0.76) (7) .
In an integrative review of the literature, the agreement between nurses and the gold standard using the MTS ranged from moderate to near perfect the MTS is applied in the studied places (14) .
A study conducted in Germany showed that agreement among nurses for the German version of the MTS was almost perfect (Kappa = 0.95) (16) . This is the Intuitive and reflexive judgment, components that involve decision-making in triage, is strongly influenced by nurses´ professional experience (2) . Corroborating this theory, in this study, the variables time of professional experience as a nurse, experience as a nurse in emergency services, and experience as a nurse in the risk classification were associated with the external and internal reliability of the MTS. In general, nurses between one and five years of experience and between five and ten years of experience obtained higher levels of agreement with each other and with the gold standard.
In Brazil, although there is an informal recommendation that the nurse should have previous experience in emergency services to act in the classification of risk, this is not a requirement regulated by the class council. In Italy, nurses are required to have at least six months´ experience in the triage patients in emergency services (17) .
Professional experience has been pointed out in the literature as a factor that influences nurses´ decision-making in triage. Nurses use previous knowledge and experience to make inferences and screen new cases (2,5,18) . The correct classification of risk depends on the training and experience of the nurse in the application of the MTS (15) .
It should be emphasized that the available literature is not conclusive about the amount of time of experience necessary to guarantee nurses´ competence in triage.
This is the first Brazilian study that investigated and found an association between professional experience and the external and internal reliability of the MTS.
In the study that evaluated nurses´ experience in the triage and the ability to use the MTS correctly, the Kappa values between nurses and the gold standard were higher when the nurse´s experience with the MTS was higher, but no significant difference was found between the categories of experience in the triage analyzed (9) . The combination of the use of the MTS, the nurse´s experience in evaluating critically ill patients, and organizational factors accounted for 65% of safety in the correct patient triage. The experience of nurses contributes to greater patient safety than the triage system, which cannot completely replace the clinical skills that the experienced nurse has developed over the years in the profession (19) . Other studies have shown that the MTS exceeds its central objective, which is to establish priority for immediate treatment (13,(20)(21) .
A study carried out showed that the sensitivity of the MTS to identify patients who died or who needed hospitalization in intensive care units ranged from 0.80 to 0.86 in adults. The specificity to measure the same outcomes ranged from 0.84 to 0.91 (13) . Patients classified as red, orange or yellow had a 4.86-fold greater risk of hospitalization, 5.58 times greater death, and a greater need to perform electrocardiogram and laboratory tests when compared to those classified as green or blue (20) . In Germany, a study showed that the higher the risk level of the patient, the longer the nursing time spent for patient care, showing that the MTS can be used to subsidize the calculation of the nursing staff in emergency services (21) .
Thus, it can be seen that the MTS is able to predict early care needs in emergency services and to direct the organization of care and management of the clinic. However, the use of a reliable instrument is important Souza CC, Chianca TCM, Cordeiro Junior W, Rausch MCP, Nascimento GFL.
for nurses´ safety, considering that the protocol is their support system in decision making.

Conclusion
The