Patient Classification System for alcohol and other drugs: construction and validation

www.ee.usp.br/reeusp Rev Esc Enferm USP · 2018;52:e03324 1 Universidade de São Paulo, Escola de Enfermagem, Departamento de Enfermagem Materno-Infantil e Psiquiátrica, São Paulo, SP, Brasil. 2 Hospital Alemão Oswaldo Cruz, São Paulo, SP, Brasil. 3 Hospital São Rafael, São Paulo, SP, Brasil. ABSTRACT Objective: To construct and validate a nursing care dependency classification system for alcohol and other drugs. Method: A psychometric study. We used the agreement percentage and intraclass correlation coefficient for inter-rater agreement. The Receiver Operating Characteristic was used to determine values and accuracy of the cut-off points. The steps were literature review, content validation and clinical validation. Results: Fifteen (15) judges, 6 nurses and 65 users of psychiatric emergency and hospitalization units participated. There were 11 indicators divided into three subcategories of dependency, and a sum between 11 and 33. The agreement regarding the judges’ evaluation on the categories was satisfactory, being equal to or higher than 80%. Agreement was satisfactory among the nurses in most of the indicators (K>0.400), and the Intraclass Correlation Coefficient was equal to 0.723. Conclusion: The Classification System of Patients for Alcohol and other Drugs presents statistical evidence of reliability, obtained by a satisfactory inter-rater agreement. Its applicability is widely believed to assist in the management of nursing care and team sizing.


INTRODUCTION
The consumption of psychoactive substances in Brazil and in the world is increasing, and its consequences are among the most frequent threats to the health of the population (1) .These mainly include traffic accidents, violence, criminality, work absenteeism, reduced performance in professional activities, difficulties in interpersonal relationships, psychological disorders, homicides, suicides, clinical diseases, and hospital admissions; moreover, such consumption represents a risk factor for morbidity and mortality among young people and adults (2) .
Treatment of dependent users of psychoactive substances aims to evaluate comorbidities and to stabilize crisis situations, with a posterior approach to psychosocial problems and requires adequate permanence time for its effectiveness (1,3) .Considering the dynamics and the susceptibility of motivational stages, users may be ambivalent and reluctant towards the actions inherent to the treatment (4) , thus requiring different amounts of nursing care (5) which implies the need for technical and scientific training of these professionals to meet these and other demands.
The Federal Nursing Council (Conselho Federal de Enfermagem) (6) emphasizes that nurses must rigorously assess health needs and plan interventions based on various parameters to ensure comprehensive and continuous care.It also points out that the number of nursing professionals directly interferes with the safety and quality of care.It defines parameters for dimensioning the nursing professional staff in order to subsidize the managers and nurses in decision making about the work process based on the Patient Classification System (PCS) (Sistema de Classificação de Pacientes -SCP) (6) .
Managing professionals in health services is based on labor standards (7) with a focus on attributes and competencies, but little consideration is given to the variability inherent in care activities and clinical aspects (8) .When there is numerical and/or qualitative insufficiency in the nursing team, injury risks to users and staff increase.Thus, a PCS contributes to establishing the amount of professionals by providing necessary work hours and reducing attendance-related iatrogenies (9) .Classifying the needs of the individual to be assisted is the first step in the nursing personnel dimensioning process, and an important administrative tool to define the complexity of care dependence in relation to the team, and can be applied in different contexts (5) .
The creation of a PCS for alcohol and other drugs is justified because of the inexistence of an instrument that evaluates the dependency of nursing care which targets users of psychoactive substances in specialized service for treatment.This PCS aims to subsidize the daily systematic evaluation performed by nurses about the nursing care needs presented by users addicted to a psychoactive substance in specialized treatment service.Building and validating this PCS will contribute to determining staff dimensioning and will enable identifying complex and multifaceted conditions among users addicted to different psychoactive substances.Thus, this study aimed to construct and validate a Patient Classification System (PCS) for alcohol and other drugs.

METHOD
This study was constituted from three stages.The first was constructing the classification system of dependence in nursing care for alcohol and other drugs based on literature review, having adults of both genders as the target population for its application.The psychometric methodology was used for its construction (10) , along with the complexity of care classification model (11) and the graphic structure according to a psychiatric nursing dependency level classification instrument (12) .
The second step of content validation was conducted in two rounds.Content validation contemplates evaluation and correction by specialists of the phenomena which is intended to be measured.Twenty (20) experts from the area of interest were invited to participate in this step, but five were excluded because they did not give a response in the first or second round.Thus, the committee consisted of 15 expert judges (13) who were selected according to criteria adapted from Fehring (14) such as experience, specialization, master's degree and/or doctorate in mental health or alcohol and other drugs.
In the first round, the judges evaluated the system consisting of 10 care indicators, judging the relevance of each indicator.After the feedback, the judges' suggestions were considered and a new indicator was added, making a total of 11 care indicators in the system.In the second round, the same group of judges re-evaluated each indicator according to the relevance, content clarity and complexity level of each subcategory.The percentage agreement was calculated by dividing the number of participants who agreed on an item by the total number of participants, multiplied by 100.A minimum agreement of 90% among the judges was adopted (15) .
The third step consisted of the validation and clinical reliability of the system which occurred in two moments: (I) System Application and (II) Statistical analysis of the data.The first one was carried out in two services specialized in treating people addicted to psychoactive substances in the city of São Paulo, from July to October 2016, including a hospitalization unit and a psychiatric emergency department.Scenario diversification was taken into account for enabling a heterogeneous sample and allows to evaluate the subcategories of the system indicators more comprehensively (15) .Six nurses who were previously trained to apply the system participated.Each user was submitted to two evaluations performed by different nurses, who were instructed not to communicate during the system application in order to avoid interference in the results.A sample of 65 users was established when considering the number of indicators that compose the system, surpassing the minimum sample of five for each indicator (16)(17) .
The second moment was performing data analysis using the Kappa Coefficient, or Cohen's kappa coefficient (K) and the Intraclass Correlation Coefficient (ICC).The first can be defined as an association measure used to describe and test the degree of inter-rater agreement.The use of Kappa made it possible to analyze the evaluation agreement among nurses.This coefficient may vary between 0 and 1, whose interpretations may be K<0 = no match; 0<K≤0.19= poor agreement; 0.20<K≤0.39= mild agreement; 0.40<K≤0.59= moderate agreement; 0.60<K≤0.79substantial agreement; 0.80<K≤1.00= almost perfect agreement (18) .www.ee.usp.br/reeuspRev Esc Enferm USP • 2018;52:e03324 The second coefficient is a measure of agreement that measures the intensity of the intraclass association in the same construct, indicates the reproducibility and the reliability of an instrument, in addition to the variability among the nurses' evaluations in the study.The ICC value also varies between 0 and 1, and the more the value approaches 1, the better the result (19,20) .
Nurses were asked to evaluate the patients according to their clinical judgment and categorize them into levels of care dependency before applying the system.These levels were pre-established taking into account concepts of Minimal, Intermediate and High dependency, which are present in the literature (21) .When considering such results as the gold-standard, the Youden index (22) was applied in order to obtain the lowest possible proportion of errors in the classification.This index determines the precision of the cut-off points, and the more its value approaches 1, the more positive its interpretation is (22) .
The ROC (Receiver Operating Characteristic) curve method was also implemented, which reveals the measure of the ability to correctly classify some given data.This method was used because it is an important tool to establish cut-off points (23) and allows to evaluate how well a condition can be correctly identified or the precision of a test, and can be interpreted as excellent (>0.9), good (0,8-0.9),regular (0.7-0.8), poor (0.6-0.7) or inadequate (0.5-0.6) (23) .The database was built in an Excel spreadsheet, and statistical analysis was performed in SPSS Statistics Base 22.0 software.All ethical aspects have been respected and approved in accordance with Opinion No. 105.395/2015.
As a limitation in the study we highlight the fact that the experimental application of the Patient Classification System for Alcohol and Other Drugs (In Portuguese: Sistema de Classificação de Pacientes em Álcool e outras Drogas -SiCAD) was carried out in a hospitalization and psychiatric emergency unit.Nevertheless, the possibility that this could be tested in other scenarios such as a psychosocial attention center is appreciated.Also, it should be pointed out that its application did not occur simultaneously in both places, which made it impossible to obtain nurses' immediate opinions on the experience of using the system.However, it was possible to circumvent this limitation by asking the nurses for concise feedback, which occurred after the data collection and analysis was finalized.It was felt that such feedback could have been more detailed if it had occurred in a more opportune moment.

RESULTS
The Patient Classification System for Alcohol and Other Drugs (SiCAD) includes 11 indicators with three subcategories (minimum, intermediate and high dependency), with a score ranging from 1 to 3 and a sum from 11 to 33, according to the complexity.Its indicators are: General Aspects; Orientation, Thinking and Sensoperception; Mood and Affection; Attitude and Posture; Behavior; Food and Hydration; Eliminations; Sleep Pattern; Pharmacological and Non-pharmacological treatment; Clinical Aspects and Motivational Aspects (this one was added after the second round).
The minimum dependency classification comprises clinically and psychologically stable users being capable of meeting most of their needs, and therefore being lowly dependent on the nursing staff.In the intermediate dependency classification, users have periods of clinical and/or psychological instability, which requires specific interventions to meet their needs and are partially dependent on the nursing team.Those classified as highly dependent are clinically and/ or psychologically unstable, present risks to themselves or others, and are largely dependent on interventions to meet their needs.
In the construct validation, 53% (n=8) of the 15 judges who participated in the study graduated between 7 and 14 years ago, and 47% (n=7) more than 18 years ago; they had postgraduate degrees in mental health, psychiatry or alcohol and other drugs, and there were 14 specialists (93%), 12 masters (67%) and three doctors (20%) among them from different parts of the country.The judges were asked about their agreement on the pertinence of the data that composed the header and the footer only in the first round, because the data obtained were satisfactory.The space designated to be filled with information about users' "Age" and "Gender", "Total system points" and "Nurse's signature" was 100%, and 93.3% (n=14) considered the "Date" and "User registration" fields to be relevant.The judges' assessment of the "Classification" of users was 86.7% (n=13).
The judges also evaluated the relevance of the 10 indicators in the same round, and there was full agreement on the indicators "mood and affection", "eliminations", "sleep pattern", "pharmacological and non-pharmacological aspects" and "clinical aspects".Disagreement indicators of 6.7% (n=1) were "general aspects", "orientation, thinking and sensoperception", "feeding and hydration", and 13.3% (n=2) with "behavior" and "interaction".
The judges questioned adopted terms and concepts, and the established interval for degrees of care dependency and clarity in the description of subcategories.After analyzing these data, changes were made in SiCAD and then submitted to the second round in which there was full agreement on the relevance of the indicators, according to Figure 1.Agreement regarding content clarity and the complexity level of each subcategory was over 80%.Six nurses participated in the clinical validation stage with a mean age of 37 (ranging from 29 to 58 years old), 5 years of Nursing practice, 3 years of experience with alcohol and drug users/mental health, and all of them with a postgraduate degree in at least one of the two specializations.
The application time of SiCAD was between 10 and 20 minutes, reducible according to the professional's familiarity with the system.The agreement analysis among the nurses showed variation among the indicators.The majority of the indicators obtained an agreement percentage over 80%.Kappa ranged from 0.160 to 0.702, showing that there was statistical significance in most of the indicators, according to Table 1.The result of the Intraclass Correlation Coefficient was 0.723, indicating reproducibility of SiCAD in clinical practice.
The complexity classification intervals of Minimum (11 -13), Intermediate (14 -20) and High (21 -33) Dependency were determined by the Youden index (22) based on the nurses' subjective evaluations, with 82% precision on the cut-off points.Sensitivity values were obtained for each complexity classification from the cross-correlation between the nurses' evaluations and the total SiCAD score: 83.2% for minimal dependency, 73.3% for intermediate and 100% for high dependency.The Quality of Adjustment Index (ROC) was used in order to validate the reliability of the determined cut-off points, resulting in 0.922.

DISCUSSION
The needs of nursing care, users of alcohol and other drugs, and adults of both genders (24) were considered in the SiCAD construction process, ranging from physiological needs to complex psychological aspects and motivational states.Dividing the indicators into subcategories allows for identifying the dependency level that the users present in relation to the nursing team and reflects the stage which they are in, as well as evidencing clinical and psychosocial manifestations (25) .The indicators that make up the SiCAD are able to provide data on the users' demands in a comprehensive way and enable planning of nursing care.
The literature identifies the participation of a group of judges as being essential in construct evaluation (26) .In this study it is recognized that the participating judges presented maturity in the area of alcohol and other drugs, allowing for a rigorous analysis of the indicators.In addition, they contributed to content adequacy when they suggested including a new indicator.Content validation may be limited by judges' subjectivity, thus an agreement percentage greater than 80% was considered acceptable in this study (15) .
When conducting the two rounds to validate the SiCAD content, an evaluation of the relevance of the concepts involved in each indicator and the extent to which they contributed to achieving the objectives proposed by the system (27) was considered important, in addition to the judges' suggestions for qualitatively improving the indicators.After applying the agreement percentage in the results of the questions with dichotomous answers (yes/no), it was possible to systematize them as to the degree of agreement between the judges' evaluations, and these results were enough to reach the objectives proposed for the content validation step (15) .
After SiCAD application, the nurses revealed having familiarity with some indicators since they had already worked with evaluation systems of specific complexity in mental health in existing clinics.The indicators that showed substantial agreement based on Kappa were: 11 -Clinical Aspects; and 1 -General Aspects, which include clinical and/or psychiatric comorbidities, signs and symptoms resulting from the use/abstinence of psychoactive substances, and appearance and fragilities in self-care, respectively.It is understood that these are common evaluations of the nurses' clinical practice.
The indicators 9 -Pharmacological/Non-pharmacological treatment and 10 -Motivational Aspects, explore drug characteristics, motivational stages, relapse risks, psychosocial aspects and adherence to treatment, and were the indicators that presented less agreement among nurses, but were well evaluated by the judges on their clarity and the increasing level of complexity.It should be emphasized that for an accurate evaluation of the user in relation to these indicators, it is fundamental that the nurses have technical-scientific knowledge and refined skills to recognize the motivational stages that imply the users' response to treatment (26) .The need for a more in-depth analysis of the interpretation of these indicators from the nurses' perspective was then identified in order to understand the possible difficulties they had at the time applying SiCAD.When using the Kappa coefficient to measure agreement among nurses' evaluations, it was possible to identify some indicators with mild agreement (1) , which is understandable when considering the differences in their training and their professional experience, as well as their familiarity in the use of ranking scales.Nevertheless, a satisfactory ICC value was obtained (19) , thus demonstrating that the SiCAD is reliable and reproducible in nursing practice.The cut-off points and intervals were statistically well-defined and support a consistent classification according to the complexity level of nursing care.
The implications for using SiCAD in nurses' clinical practice can be demonstrated both from the viewpoint of care and management.In the first point, using SiCAD allows for evaluating the real care needs of the users and favors elaborating the evolution and implementation of the Systematization of Nursing Care (SNC).In the second point, as a management tool it maps the population profile that was provided care in order to lead organizational and/or adaptations changes, it identifies weaknesses and points for improvements in order to direct the teams' actions of continuing education.In addition, from a longitudinal perspective it identifies the complexity of the required care and sustains discussions regarding the quantitative aspect of human resources.

CONCLUSION
This study enabled constructing and validating a Classification System of Care Dependence for Alcohol and Other Drugs, abbreviated as SiCAD.It presents psychometric indicators with acceptable content validity, as well as reliable statistical evidence obtained by satisfactory inter-rater agreement.The classification system intervals are reliable and accurate since they have been statistically adjusted, which is an essential differential and makes them more sensitive to correctly categorize user needs.
SiCAD application is private to the nurse and enables daily care need assessment of users who are dependent on psychoactive substances inserted in a specialized service for treatment.The system shows the attended population's profile and contributes to the adequacy of nursing team dimensioning.
SiCAD proved to be easy to use, applicable in a short period of time and encompasses several aspects that are the focus of nursing care.It is also recognized that its effectiveness can be evidenced from its application in clinical practice.We believe in its importance, emphasizing studies that deepen the use of this technology in nursing care.

Figure 1 -
Figure 1 -Percentage of inter-rater agreement among judges regarding the pertinence of the first and second round indicators -São Paulo, SP, Brazil, 2016.

Table 1 -
Agreement percentage distribution and Kappa coefficient for the indicators based on the nurses' evaluations -São Paulo, SP, Brazil, 2016.