Accuracy of a pediatric early warning score in the recognition of clinical deterioration

ABSTRACT Objective: to evaluate the accuracy of the version of the Brighton Pediatric Early Warning Score translated and adapted for the Brazilian context, in the recognition of clinical deterioration. Method: a diagnostic test study to measure the accuracy of the Brighton Pediatric Early Warning Score for the Brazilian context, in relation to a reference standard. The sample consisted of 271 children, aged 0 to 10 years, blindly evaluated by a nurse and a physician, specialists in pediatrics, with interval of 5 to 10 minutes between the evaluations, for the application of the Brighton Pediatric Early Warning Score for the Brazilian context and of the reference standard. The data were processed and analyzed using the Statistical Package for the Social Sciences and VassarStats.net programs. The performance of the Brighton Pediatric Early Warning Score for the Brazilian context was evaluated through the indicators of sensitivity, specificity, predictive values, area under the ROC curve, likelihood ratios and post-test probability. Results: the Brighton Pediatric Early Warning Score for the Brazilian context showed sensitivity of 73.9%, specificity of 95.5%, positive predictive value of 73.3%, negative predictive value of 94.7%, area under Receiver Operating Characteristic Curve of 91.9% and the positive post-test probability was 80%. Conclusion: the Brighton Pediatric Early Warning Score for the Brazilian context, presented good performance, considered valid for the recognition of clinical deterioration warning signs of the children studied.


Reference standard and cut-off point of the BPEWS-Br for clinical deterioration
Diagnostic test studies need a gold/reference standard that establishes the presence or absence of a disease/event. When it is not possible to determine a gold standard, clinical criteria based on the history and physical examination can be used to establish a diagnosis (16) .
In studies that validate pediatric early warning scores, certain authors have reported difficulty in establishing a reference standard for clinical deterioration in children (8,10,17) . Some of these have used the call for the Rapid Response Team (RRT) (11) , while others have adopted the transfer to the Intensive Care Unit (ICU), however, they recommended that more standards should be tested (10,18) .
In this study, considering that a PEWS aims for the early identification of signs of clinical deterioration; that there is no consensus reference standard for this event; that there is a shortage of pediatric ICU beds in the municipality and a lack of an RRT in the study scenario, the classification of children "without signs of deterioration" and "with signs of deterioration" was made guided by a set of criteria based on the Primary Clinical Evaluation of the Severely Ill Child, recommended by the American Heart Association (AHA) and the American Academy of Pediatrics (AAP) (19) .

Among the criteria of the Primary Clinical
Evaluation of the Severely Ill Child, blood pressure was excluded, because it was a late sign of cardiovascular decompensation in the child, as were the Glasgow Coma Scale and the pupillary reaction, opting for the use of the AVPU Pediatric Response Scale (Alert, Responds to voice, Responds to pain and Unresponsive) for rapid neurological assessment (19) .
From a broad discussion among the researchers of this study regarding the reference standard adopted, it was defined that 3 or more altered clinical signs in the primary clinical evaluation would classify the child as "with signs of deterioration".
Regarding the BPEWS-Br, the score to trigger deterioration was defined by the best cut-off point obtained by the ROC curve. The BPEWS-Br ≥3 was able to maximize sensitivity and specificity and obtained excellent accuracy. Thus, children with a final score <2 were considered "without warning" and those ≥3 "with warning signs for clinical deterioration". Exclusion criteria were children with medical discharge prescribed, hospitalized in the cardiology or oncology units and those with precautionary measures. Children with heart disease were excluded because there is already a validated warning score for this population in the literature (20) . Oncology children were excluded due For each day of data collection, one unit was drawn, and the children admitted to that unit, who fulfilled the inclusion criteria and ethical criteria, participated in the study regardless of whether or not they showed signs of clinical deterioration, considering that in diagnostic test studies it is necessary to have sick and healthy patients.

Data collection
Three instruments were used in the collection:

Ethical issues
The parents/guardians signed the consent form,   Table 3 shows the validity indicators of the BPEWS-Br applied to the population studied for scores ≥3 and ≥4. The higher score produced lower sensitivity and NPV and higher specificity, PPV and likelihood ratios.

ROC curve
According to Figure

Discussion
The validity indicators obtained in this study showed that, based on the reference standard adopted, the BPEWS-Br proved to be a valid tool, with good performance in the indication of warning signs for clinical deterioration in the children studied, increasing the probability of this event occurring when the score was ≥3.

Some important aspects of the studies that sought
to validate the BPEWS in its original, adapted or modified versions need to be analyzed, discussed and compared with data from the present study, such as the various indicators/reference standards for clinical deterioration in children, the cut-off points that indicate the event of deterioration, the validity indicators calculated for the score, the scenarios, the samples and the age groups of the children to whom the score was applied.
Among others, the following reference standards for identification of clinical deterioration have been used to verify the validity of the BPEWS, transfer to the ICU (10,12,18,(22)(23) ; call for the RRT; Code Blue -CB (called before cardiorespiratory arrest) (11) ; and admission into the hospital (22,24) . In this study, none of these standards were used, choosing instead a set of criteria based on the Primary Clinical Evaluation of the Severely Ill Child guided by the AHA and AAP (19) .
The above criteria were followed in order to verify the validity of the BPEWS-Br regarding its actual aim, which is to assist the health team in the early recognition had a score ≥3 (10) , values that were very different.
Regarding the cut-off point of the BPEWS, in order to indicate clinical deterioration, some studies considered or found varied scores: 1 (12,22) , 2 (12,18) , 2,5 (23) , 3 (10-11) e 4 (11,24) . The author of the BPEWS advised that a final score of 4 or a score of 3 in one of the partial components should trigger the call for the RRT, characterizing the clinical deterioration event. However, this behavior could be adapted according to each scenario (6) .
It is necessary to consider that the more the cut-off point is reduced, the greater the sensitivity and the lower the specificity of the score; Thus, healthy patients can be identified as ill by the test (false positives). The ideal is to strike a balance between sensitivity and specificity.
In this study, the BPEWS-Br score of 3 was the cut-off point that maximized sensitivity (73.9%) and specificity (95.5%) and obtained the best accuracy (91.9%).
In order to evaluate the performance of the BPEWS, the sensitivity, specificity, predictive values and the areas under the ROC curve were calculated in the majority of the studies (10)(11)(12)18,(22)(23)(24) to obtain the accuracy of the score, with varying results. In some studies, the likelihood ratios (12,22) were calculated; The post-test probability, calculated in this study, was not found in any of the studies analyzed.
The likelihood ratio has been an innovative and useful concept in studies of diagnostic accuracy. When multiplied by the pre-test probability, the LR+ and LRwill generate the post-test probabilities, indicating how much the test result will increase or decrease the pretest probability of a disease (21) , hence its importance.
Thus, the PEWS were not constructed as indicators of emergency situations or of admission to the ICU or the hospital, which imposes certain limits on their use.
It is important to note that, depending on the reference standard and cut-off points of the BPEWS, the prevalence of clinical deterioration, as well as performance indicators of the score, may vary and influence the study results.
Regarding the study scenarios, the BPEWS was conceived as an warning instrument for children hospitalized on wards (6) , where urgency and emergency situations are not part of the daily routine of the health team. Therefore, this is a score that can contribute as a support instrument for these teams in the recognition of the clinical severity of the patient. Thus, the majority of the study scenarios for validation of the BPEWS were performed on wards (10)(11)18,23) , however, some authors also applied the score in the emergency unit, upon arrival of the patients (12,22,24) .
For this study, the scenarios used were the clinicalsurgical wards and emergency observation/stabilization units, where the patients would already be hospitalized.
The emergency units were included as they are places where clinical deterioration is more common when compared to the wards, since, in diagnostic test studies,