The effectiveness of substance abuse treatment : development of a brief questionnaire

Background: Practitioners need brief instruments to monitor outcomes in both treatment of drugs and alcohol addiction because they are useful to guide decision making in a short time. Objectives: This study aims to develop a brief questionnaire, based on Client Evaluation of Self and Treatment, to evaluate the treatment effectiveness in drug and alcohol addiction treatment settings. Methods: A cross-sectional study using a convenience sample (N = 608) recruited from Division for Intervention on Addictive Behaviours and Dependencies (DICAD – ARS North). Results: The results show a new four-factor solution that accounted for 54.4% of the total variance and that provides the best fit to the data (c2/df = 1.72, CFI = .94, GFI = .91, RMSEA = .048 [.040-.057]; prmsea = .623). It also revealed a high internal consistency (a = .82). It was found a significant negative correlation (r = .52, p < .01) between the final version of the instrument and a self report measure of psychopathology symptoms. Discussion: This brief questionnaire, with good psychometric properties, can be useful to provide a viable and rapid feedback of treatment outcomes. Further studies should be performed to continue the evaluation of the reliability of this measure. Moura A et al. / Arch Clin Psychiatry. 2015;42(4):83-9


Introduction
There have been significant changes in addiction treatment over the past three decades, where has been a significant movement from inpatient to outpatient programs.However, while treatment programs have changed over time, monitoring the effectiveness during the implementation of these programs has been little explored both by researchers and practitioners 1 .There are two underlying main approaches to conceptualize the treatment effectiveness.One is the rehabilitation-oriented model is the traditional paradigm where the method used to evaluate the effectiveness of addiction treatment is based on post-treatment follow-up outcomes, assuming that benefits were attained during treatment, and positive changes have occurred by the end of the treatment, and it is expected sustained abstinence at the time of follow-up 2 .
This method has been used in the case of residential treatment, where drugs are not available in the treatment setting, and the effectiveness outcome is evaluated for a substantial follow-up period of time.However, the limitation of this method is the excessive attention to the sustained and complete recovery from a substance use as criterion of treatment effectiveness.According to White 3 the problem is that "groups like Narcotics Anonymous (NA) have defined recovery in terms of abstinence from drug use, but addiction scientists have generally defined recovery from illicit drug dependence in terms of problem resolution rather than absence of drug use".Thus, this conceptualization "would allow measuring levels of outcomes over time and answer questions about the viability of particular problemresolution strategies for particular populations" 3 .In addition, the pos-treatment follow-up method has other several limitations.The evaluations are usually conducted by an external researcher from the treatment team 2 , who do not measured patient changes during treatment.When participants are not located or refuse to participate at follow-up evaluations, the internal and external validity of the collected data could be compromised due to the characteristics of those participants who drop out (e.g., involved in antisocial and criminal behaviors) 4 .Finally, many complex ethical issues arise from the follow-up evaluations, in part, due to the assertive methods required to generate high follow-up rates 5 .
Other approach that has been suggested by researchers is the alternative method of evaluating the treatment effectiveness based on the monitoring method instead of follow-up results.This new conceptualization was emerged from two changes, namely the change of the health-care delivery system that moved from residential care to outpatient setting, where nowadays practitioner's need for more economical, rapid and clinically relevant information to guide decision making, and the transition of the recovery definition from medicalized term to a problem-solving process, and a psychosocial perspective 2,3 .The proposal called "concurrent recovery monitoring" (CRM) 2 could be described as a brief and repeated evaluations, relatively easy to collect, concurrent with treatment to monitor and assist patient change, toward clinical and social outcomes, and more relevant and in time information to guide the decision making in a more effective way 2 .For instance, clinical decisions about which specific treatments are more suitable according to the different patterns of drug or alcohol use and the different characteristics of drug users 6 .Monitoring can also optimize the results of treatment 7 and monitor the need and the readiness for change in programs and organizational factors 8 and contribute to retention in treatment 9 .Further, the monitoring method can overcome the ethical problems of complex and large assessments, including the burden of time and efforts to generate high follow-up rates 5 .However, the practitioners need economical, rapid and clinical reliable methods to guide decision making in a short time period without compromising the patient attendance and participation in treatment.Complex or lengthy assessment measures may also lead to staff noncompliance 10 .On the other hand, the monitoring system can be burdensome for clinical staff in cases of large time-consuming intake, weekly and follow-up assessments, and can compromise the quality of data through a monitoring system 11 .The solution appears to be the use of self-report questionnaires that took a short-time to be completed, not requiring the attendance of the staff member in training program, as in the case of interviews, and are well appropriate to systems of care 10 .
Machado et al. 1 conducted a study with individuals undergoing dependence treatment, where they created a software that allows building a database for monitoring the intervention efficacy in clinical practice.For that, the authors used the following instruments: (1) Drug Abuse Screening Test -Portuguese version (DAST) -a dichotomous 20-item scale to assess the severity level of consumption.It evaluates the consequences related to consumption during the last 12 months such as physical and psychological symptoms, social and relational aspects, among others; (2) Outcome Questionnaire -Portuguese version (Q-45), a 45-item questionnaire that provides a reliable assessment of various aspects of the adjustment level and psychosocial disturbance of individuals; (3) The Inventory of Drug-Taking Situations -Portuguese version (IDTS) -an instrument consisting of 50 items that describes potential drug use situations over the past year; it allows to recognize problematic situations and anticipate risk situations; (4) Consumer Satisfaction Questionnaire -Portuguese version (CSQ-8), an 8-item instrument to explore the degree of patient satisfaction regarding to treatments and services received and their impact on the patient's life.This study brings a clear progress in terms of research in this area in Portugal.
However, the administration of this wide array of assessment tools leads to a final database of about 123 items, making it difficult to apply on a regular basis in the clinical setting.On the other hand, if we choose not to apply the same instruments in all stages of evaluation/monitoring we are not able to compare results of the various moments of administration, compromising the effective therapy evolution of individual consumers during their treatment.
There are other instruments available to monitor the effectiveness of treatment programs.Marsden et al. 12 validated, for the European context, the Treatment Perceptions Questionnaire (TPQ).They also validated the Maudsley Addiction Profile (MAP), which adds some of the most relevant performance indicators in the evaluation of treatment outcomes.Notwithstanding, other important indicators of effectiveness (including the therapeutic relationship, the involvement or active participation in treatment, the social support network 7,[13][14][15] , psychopathological symptoms, negative affects, or treatment orientation [16][17][18][19] are missing in this questionnaire. Recently, studies conducted by Joe et al. 7 and Simpson 14 validated the Texas Christian University (TCU) Client Evaluation of Self and Treatment (CEST) a multidimensional instrument covering the main indicators of effectiveness of drug and dependence treatments that can and should be administered in repeated evaluations during the treatment process.The CEST was developed under the DATAR project, founded by the National Institute on Drug Abuse in 1993 (Grant No.DA13093).The CEST has originally 130 items divided into 17 dimensions, taking 30 to 40 minutes to complete, which is its biggest limitation.
So, we aimed to develop a short self-report measure, suitable for both inpatient and outpatient programs, encompassing evaluating of critical indicators of treatment effectiveness, using a problemsolving and a psychosocial models.For that, we selected the previous Portuguese version 20 of Texas Christian University -Client Evaluation of Self and Treatment -TCU -CEST.We also aimed to examine the validity of the new instrument comparing it with a psychopathology self-report measure, similarly to previous studies of the TCU-CEST 7,14 .

Participants
Six hundred and eight subjects were recruited from Intervention Service on Addictive Behaviours and Substance Dependence (SICAD) -Regional Northern Section, in Portugal.The convenience sample was recruited from 3 main treatment settings: outpatient's drug addiction treatment (64%; n = 389); outpatient´s alcohol dependence treatment (14%; n = 85), and inpatient's drug and alcohol dependence treatment (22%; n = 134).
We selected patients who were in treatment for, at least, one month.Furthermore, the researchers took into account changes resulting from psychopathology or recent consumption, which could interfere with the normal completion of the questionnaires

Procedures
After we obtained permission from the authors to use The TCU-CEST, SICAD and DICAD, then the study was submitted to the ethics committee of the Faculty of Psychology and Education Sciences (University of Porto), which approved the study.
Researchers certified on addiction problems administered the instruments at the facilities of DICAD -North Regional Health Administration after the participants signed the free consent inform about the research.The confidentiality was ensured to all participants.

Materials
Socio-Demographic Questionnaire.This questionnaire included information about birth date, gender, age, birthplace, treatment program, treatment time and finally, the identification of primary substance of abuse.
Brief Symptoms Inventory (BSI) 21,22 .This is a reduced version of the Symptom Check List 90 Revised (SCL-90-R), a self-report inventory consisting of 53 items.The participant must specify the degree to which each problem has affected them over the past week, on a Likert-type scale (1 -Never, 2 -Rarely, 3 -Sometimes, 4 -Often, 5 -Very often).This inventory assesses psychopathological symptoms along nine basic dimensions (somatization, obsession-compulsion, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation and psychoticism) and three global indices (positive symptoms distress index -PSDI, global severity index -GSI, and positive symptom total -PST).In the present study, the various dimensions have an internal consistency ranging from moderate to high, with Cronbach's alpha values between .70 to .85.The internal consistency of the overall BSI in this sample was .96.
Texas Christian University (TCU) -Client Evaluation of Self and Treatment (CEST) 7,14 .This instrument was developed as part of National Institute on Drug Abuse (NIDA).The original CEST is a 129-item self-rating instrument that includes 17 scales measuring patient functioning and treatment perceptions.Psychometric properties (including reliability and construct validity) of the scales are examined in Joe et al. 7 and acceptable reliabilities (.70 or above) were generally reported, and construct validity was also demonstrated.The response format is a 7-point Likert scale (1 -I strongly disagree to 7 -I strongly agree).We used the Portuguese version 20 that included 101 items that match the original 4 dimensions and 13 subscales, with reasonable to good internal consistency values (ranging from .

Statistic analysis
We used the Statistical Package for Social Sciences -SPSS (SPSS Inc., Chicago, IL, version 19.0) to perform the analyses.Concerning the psychometric sensitivity of the items, we found a high kurtosis (|ku| > 7) for item 62 and a high skewness (|sk| > 2) for items 8, 15, 26, 54, 57, 62, 79, 104 and 127 23 .Without these items, we sought to establish the factor validity of the questionnaire through the Exploratory Factor Analysis (EFA), using the sample of 296 patients in treatment.In conducting the EFA, we used for factor extraction the principal axis factoring method and oblimin rotation with Kaiser normalization.Following the recommendations of Dziuban and Shirkey 24 , before the analysis, we explored the psychometric adequacy of the items.Next, we performed a confirmatory factor analysis through AMOS software.We also screened for major violations of normality, taking into account the absolute values of kurtosis (ku < 7) and skewness (sk < 2) 23 .The hypothesized model obtained through the EFA was evaluated using indices and respective values for an acceptable fit 25 : c 2 /df < 5 26 Comparative Fit Index -CFI > 0.90; Goodness Fit Index -GFI > 0.90 27 Standardized Root Mean Square Residual -SRMR < 0.10 28 , and Root Mean Square Error of Approximation -RMSEA < 0.08 26 .A 90% confidence interval (Low90 and Hi90) was used to assess the precision of the RMSEA estimate 29 .We also considered p Close Fit -prmsea > 0.50 29 .

Results
Regarding the sample, participants from outpatient´s drug addiction treatment included mainly men (89.4%), with ages ranging from 18 to 61 (M = 39.7;SD = 7.8).The psychoactive substance reported as the primary drug of abuse prior to treatment was heroin for 58% of the sample, followed by poly-consumption (19.8%), cocaine (19.3%), hashish (2.1%) and replacement psychotherapeutic drugs -methadone/Subutex -for 0.3% of this sample.The mean duration of treatment at the time of questionnaire administration was 40.61 months (SD = 40.9).The sample collected in outpatient's alcohol dependence treatment included mostly men (82.4%), with ages ranging from 28 to 73, (M = 47.3;SD = 9.4).The psychoactive substance reported as the primary drug of abuse prior to treatment was alcohol for 100% of the sample.The mean duration of treatment at the time of questionnaire administration was 32.1 months (SD = 39.9).Finally, the inpatient's drug and alcohol dependence treatment sample included mainly men (90.1%), with ages ranging from 18 to 60 (M = 39.7;SD = 8.5).The psychoactive substance reported as the primary drug of abuse prior to treatment was alcohol for 56.4% of the sample, followed by heroine (18.8%), cocaine (9.8%), psychotherapeutic drugs -methadone/Subutex (9%), poly-consumption (4.6%), and other unspecified psychotherapeutic drugs (1.5%).The mean duration of treatment at the time of questionnaire administration was 2.6 months (SD = 8.9).Further, the total sample of 608 participants was randomly divided into two subsamples (N = 296 and N = 312) to properly perform the exploratory and confirmatory factor analyses.

Exploratory factor analysis
Bartlett's Test of Sphericity was significant at p = .000,indicating a good correlation between the variables and the adequacy of the EFA to this scale.The Kaiser-Meyer-Olkin index was higher than the recommended value of .50(KMO = .822).This last value showed us that the results obtained can be considered good according to Marôco 30 and Pestana and Gageiro 31 .Communality coefficients (h 2 > .40)were also taken into account to assess the psychometric qualities of the instruments 32 .Nevertheless, we chose to keep the items with communalities above .30if they are theoretically justified.According to Cattell's scree plot graphic 31 , there are four to six main factors.We ran EFA, establishing 4 factors because the theoretical model of the original version also showed 4 overall factors 7 .The four-factor solution accounted for 29.9% of the total variance.However, some items had low communality values (h 2 < .40).In this process, we eliminated the items with poor results (h 2 < .40 if they are not theoretically justified) and ran the EFA repeatedly until we found a final model in which all items showed adequate results (Table 1).In this four-factor solution all items presented communality values above .35and factor loadings ranging between .49and .90(Table 1).The final EFA revealed a new four-factor solution that accounted for 54.4% of the total variance.We renamed some of the subscales since there is a new combination of items: Therapeutic involvement (26.6%);Negative affects (12.4%);Social support (9.2%); and Peer support (6.2%).

Confirmatory factor analysis
All items were distributed within the normality range (|ku| < 4.601 and |sk| < 1.899).Sixteen outliers were found using Mahalanobis squared distance (p1 and p2 < 0.00) 25 .Because the analysis of the model fit without outliers showed similar results, we decided not exclude these cases from the analysis 30 .The model obtained by the EFA with 4 factors and 28 items was tested (Figure 1).The hypothesized model revealed a poor global fit (Table 2).However, we have analyzed modification indices and we proceeded to the change of paths only when if it made sense according to the theoretical premises: the regression values show us that items 20, 21 29, 43, 77 and 99 are repeatedly associated with items outside the factor they belong to, thus revealing their multidimensionality (see contents of the items in attachment file).Consequently, we decided to remove these items from the model.The modification indices also suggested covariance between the errors of items 74 ↔ 76, 41 ↔ 92 (see contents of the items in attachment file).Following these changes, we obtained a short version with 22 items.For this new version, results allow us to classify the model (Figure 2) as a whole as acceptable/good (c 2 / df = 1.724,CFI = .941,GFI = .907,SRMR = .155,RMSEA = .048[.040 -.057]; prmsea = .623) 33.

Reliability (Cronbach's alpha)
According to the recommended default value of > .70 31

Convergent validity
Correlation coefficients are reported in tables 3 e 4. The results show that there is a positive and significant association between all subscales and the total score of the new instrument (ranging from r = .45to r = .68,p < .01),except for the subscale Negative affects, whose association is significant, but negative, as expected by theory (r = -.74 , p < .01 ) (Table 3).We also found a significant negative association between the total score of the new instrument and the General Symptom Index of BSI (BSI-GSI) (r = -.52,p < .01),as well as between the total score of the new instrument and all BSI subscales (p < .01)(Table 4).In turn, the subscales Negative affects and Social support are significantly associated not only with the BSI-GSI (r = .59,p < .01 and r = -.26,p < .01),but with all dimensions of the BSI as well (p < .01)(Table 4).

Discussion
The main goal of this article was to develop a brief questionnaire of treatment monitoring, suitable for both inpatient and outpatient programs, to assess critical indicators of treatment effectiveness in a short time period.The choice of the TCU-CEST instrument was based upon four basic criteria: (a) adequate psychometric proprieties found in the original instrument, (b) several TCU-CEST subscales have been applied to a variety of dependence treatment programs, (c) wide assessment of several efficacy indicators, and (d) confirmed usefulness in clinical practice and in the present changes in treatment programs 7,14 .We started from a theoretical model of this original CEST 7,14 , and the Portuguese version of Moura et al. 20 , to found a brief questionnaire to evaluate the treatment effectiveness of substance abuse, using both inpatient and outpatient samples, and subsequently EFA and CFA analyses.According to our purposes, we found a different factor structure when compared to the original instrument, with a four-factor model from EFA analyses.These four factors also showed adequate internal reliability.The adequacy of this structure to a different sample was confirmed through the CFA.Nevertheless, a model re-specification was necessary.We believe that the statistical model allowed us to find a different version due to the inclusion of patient characteristics of different treatment settings, compared to the original instrument.Despite this, our final model covers the main treatment effectiveness areas pointed out in literature.The first factor, identified as Therapeutic involvement dimension, includes items intended to measure treatment engagement, treatment satisfaction and counseling rapport.The second factor, identified as Negative affects dimension, includes items to measure the psychological functioning in terms of depression, hostility and anxiety.Third and fourth factors, originally identified as Social and Peer support scales, respectively, seek to measure social and contextual factors that also affect recovery process dynamics 7 .Former instruments are extensive, which takes a long time to be completed and analyzed, like the software developed by Machado et al. 1 , where several instruments were combined resulting in a 126-items instruments.Besides, these instruments don't include some important constructs stressed by literature such as social and peer support network 7,14 , the therapeutic relationship, the involvement or active participation in treatment [13][14][15] , psychopathological symptoms or negative affects [16][17][18][19] .
In this way, when compared with others instruments, this new short-form presented here is a proper response since: (a) it is suitable for both inpatient and outpatient programs; (b) encompasses few, but critical indicators of treatment, using a problem-solving and a psychosocial models; (c) takes short time to be completed; and (d) allows building a database for the monitoring of intervention efficacy available in clinical practice.
Our second aim was to investigate the convergent validity of the brief instrument.There were significant associations of most of these subscales with each other and with the total score, suggesting the concordance of the subscales for the evaluation of different aspects of a common construct.Furthermore, we concluded that this study showed a significant correlation between the brief instrument and the BSI.The majority of the brief instrument subscales (75%) were significantly associated with at least five or more dimensions of the BSI, and half of the brief instrument subscales (50%) were significantly associated with all dimensions of the BSI and total score.Furthermore, the global score of the brief instrument was also significantly associated with all dimensions of the BSI and total score.Exceptionally, the scale Peer support was not significantly associated with any of the BSI's dimensions.Despite of the low probability that this subscale had any relation with psychopathological symptoms, it was significantly associated with the other subscales of the new instrument, as well as with the total score, being reasonable to keep this scale in the final instrument.
These results are consistent with the literature stating that psychopathology 34,35 and comorbidity 36,37 are very commons in this population.Psychopathology was one of the best predictors of the effectiveness of treatment programs 35 , increasing the likelihood of treatment failure 34,35 .Therefore, if this brief instrument is strongly and significantly associated with the BSI (a predictor of the effectiveness of treatment programs), these data seem to reinforce the potential of this version in the estimation of treatment efficacy.Considering the practical implications, this new brief questionnaire, with adequate psychometric properties, can be useful to provide a viable and rapid feedback instrument for counselors, treatment programs and policy makers, covering some of the main and most relevant areas related to the monitoring of drugs and alcohol treatment programs 3,5,[7][8][9] .Furthermore, this practice can also enable the adjustment of the interventions according to the users' needs and comorbidities 7,9,14,20 .We also suggest that this instrument should be part of the daily work in clinical practice in Portugal.Thus, more studies may be conducted in context: measuring the treatment effectiveness and warning of the need for effective changes in the recovery process.

Limitations
First, these three major treatment settings are not necessarily representative of all treatment programs for drug or alcohol addiction in Portugal.Second, this short version does not measure the same number of dimensions of treatment efficacy as the original version.Thus, we can conceive it only as an important complementary tool for this assessment.Finally, more studies should be conducted in Portugal with this short version to assess temporal validation of this instrument, and to observe correlations with others indicators of treatment progress (such as attendance, time in treatment or completion of treatment) in order to underline its predictive capacity.

Table 1 .
Final Exploratory Factor Analysis: communalities and factor loading by item (N = 296) Note: Boldface indicates the items belonging to the factor.