Abstract
Preoperative evaluation in elective surgeries has been associated with successful surgical treatment. However, there is no solid scientific evidence that screening for coronary artery disease (CAD) reduces surgical risk. The aims of this study were to describe the frequency of inappropriate investigation of obstructive CAD induced by pre-anesthetic assessment in individuals without cardiovascular symptoms (candidates for low- to intermediate-risk surgeries) and to evaluate predictors of this conduct. We performed a retrospective evaluation of medical records of anesthesiology services from patients undergoing pre-anesthesia assessment between May 2015 and May 2016, including those with functional capacity ≥4 metabolic equivalents without a diagnosis of heart disease. A total of 778 medical records (47±16 years of age, 62.6% female) were studied. A private hospital performed 50.1% of the surgeries and 60.4% were of intermediate risk. Only 2.7% (95%CI: 1.7-4.1%) were screened for CAD, and 91% of these requests were mediated by cardiology consultations performed during pre-anesthetic testing visits. Factors associated with screening for CAD were hypertension, diabetes, moderate systemic disease (ASA III), cardiac consultation, previous diagnosis of CAD, and admission to a private hospital. Independent predictors were private hospitals (OR: 3.9; 95%CI: 1.3-11.0), ASA III (OR: 5.3; 95%CI: 1.7-16.2), and hypertension (OR: 3.8; 95%CI: 1.5-9.8). The frequency of inappropriate requests for CAD screening in asymptomatic individuals without untreated systemic diseases was low in pre-anesthetic visits. Although infrequent, screening for CAD is more common in the private setting, in patients with poorer health status, and is usually prescribed during cardiology consultation.
Preoperative period; Perioperative care; Routine diagnostic tests; Operative surgical procedures
Introduction
Data recorded in the information system of the Brazilian Unified Health System (SUS) show that throughout 2019, approximately four million patients underwent surgical procedures in Brazil, with elective surgeries accounting for more than 2.5 million of them (1). Therefore, given the high prevalence of heart disease in the general population, almost a third of these patients have either coronary artery disease (CAD) or CAD risk factors (2). The prevalence of clinical CAD is around 10% (3) in developed or developing countries, and the incidence and severity of CAD increases with age in both sexes, increasing the frequency of surgeries (4 –6). Thus, the incidence of adverse cardiac events, such as acute myocardial infarction and death from cardiac causes after non-cardiac surgery may reach 3.9%, particularly in major surgeries (7,8).
Pre-anesthetic evaluation aims to minimize the risk of perioperative surgical complications, by planning intraoperative trans-care to prevent cardiovascular events associated with hypotension, volume overload, and acute anemia (8). A common practice emerging from these evaluations is the pursuit of the diagnosis of silent coronary disease (screening), motivated by the belief that this would generate cardiovascular protection for patients. Although this approach generates a sense of safety in patients with negative results, positive results promote a cascade of conduct that not only fail to attenuate the risk of cardiovascular events (7,9,10), but may lead to undesirable outcomes, ranging from suspension of surgery to inappropriate revascularization and adverse events resulting from overdiagnosis and overtreatment (11). Therefore, surgeons and anesthesiologists often consult with cardiologists during this assessment to reduce serious cardiac events (12).
Thus, CAD screening is deemed inappropriate in minor and medium complexity elective surgeries when patients are asymptomatic and have at least moderate functional capacity and eventual systemic diseases under clinical control (13,14). The screening does not effectively contribute to preoperative evaluation nor reduces the risk of cardiac complications in the perioperative period and may be potentially harmful to the patients' physical and mental health (14–16).
Therefore, the present study aimed to describe the frequency of inappropriate investigation of obstructive CAD induced by pre-anesthetic assessment in individuals without any cardiovascular symptoms, candidates for minor and medium complexity non-cardiac surgeries, and to evaluate potential outcomes of this approach.
Material and Methods
Study design
This was an observational, cross-sectional study, with data collected from medical records of the Hospital Universitário Professor Edgard Santos (public) and Hospital Jorge Valente (private, with no teaching activity), both in Salvador, Bahia, Brazil.
This research was approved by the Research Ethics Committee (CEP) of the Escola Bahiana de Medicina e Saúde Pública (EBMSP) under CAAE 57161016.8.0000.5544 and by the CEP of the Hospital Universitário Professor Edgard Santos (HUPES) under CAAE 57161016.8.3001.0049. Due to the retrospective nature of the study, both CEPs waived the informed consent form.
Sample selection
Medical records of patients who underwent pre-anesthetic consultation between May 2015 and May 2016 were analyzed. The following inclusion criteria were considered: candidates for low- to intermediate-risk surgeries, asymptomatic for cardiovascular diseases, functional capacity ≥4 metabolic equivalents (MET), and with no history of untreated systemic diseases.
Endoscopic procedures, superficial procedures, cataract, breast, or outpatient surgeries were classified as minor elective surgeries (risk of perioperative mortality <1%) (17). Intraperitoneal, intrathoracic, carotid endarterectomy, head and neck, orthopedic, and prostate surgeries were classified as medium complexity surgeries (risk of perioperative mortality between 1 and 5%).
In the two anesthesiology services analyzed, medical records provided information on functional capacity in a parameterized manner according to the Duke Activity Status Index (DASI), a questionnaire with 12 items assessing daily activities, such as personal hygiene, locomotion, domestic tasks, sexual function, and recreation with the respective metabolic costs (18). Each item carries a specific weight based on the metabolic cost. Participants are asked to identify which of the activities listed they can perform. The final DASI score varies between zero and 58 points, and the higher the score, the better the functional capacity. It is worth emphasizing that positive responses to the first four DASI items estimate functional capacity at more than 4 MET.
We excluded medical records lacking relevant information on pre-anesthetic evaluations, return visits with other doctors (inter-consultations), and reports of complementary exams.
Data collection
Data collection was performed by reviewing medical records. The main variable analyzed was obstructive CAD screening as part of preoperative assessment, defined by the following tests: exercise stress test, stress myocardial perfusion scintigraphy, echocardiogram under pharmacological or physical stress, coronary angiotomography, or coronary angiography.
Clinical and anthropometric data and physical status according to the American Society of Anesthesiology (ASA) (19) were collected and analyzed, along with high-risk comorbidities for obstructive CAD, surgical topography, and surgical extent, and the type of hospital (public or private).
Data analysis
The data obtained were double entered into the SPSS computer program (IBM, version 16.0.2, 2008, USA), with verification of consistency and range. Descriptive statistical analysis was performed with categorical variables presented as absolute and relative frequencies, while continuous variables are reported as means±SD after data normality was determined by the Kolmogorov-Smirnov test. Alanine aminotransferase and creatinine values are reported as median (first quartile [1Q]-third quartile [3Q]).
The frequency of CAD screening was described in percentage and inaccuracy was quantified by a 95% confidence interval (CI). In the univariate analysis, individuals with and without CAD screening were compared regarding numerical variables using Student's t-test, and for categorical variables using Pearson's chi-squared test (χ2). Variables with P≤0.20 in the univariate analysis were selected for multivariate analysis by logistic regression, performed by the stepwise technique, to identify predictors of inappropriate CAD screening. Unadjusted and adjusted odds ratios (OR) and 95%CI were calculated. Statistical analyzes were two-tailed, and statistical significance was defined by P<0.05.
Results
Sample features
A total of 800 medical records were screened, of which 10 (1.2%) were excluded due to lack of a pre-anesthetic evaluation form, eight (1.0%) due to the absence of an inter-consultation report, while four (0.5%) records did not have the complementary examination reports requested. Thus, 778 medical records of asymptomatic patients who were candidates for non-cardiac surgeries and had undergone pre-anesthetic evaluation were analyzed. Patients’ age was 47±16 years, 63% women, 60% underwent medium complexity surgeries, and 50% were recruited from a private hospital. Other characteristics are described in Table 1.
Mean body mass index (BMI) was 27±5 kg/m2, mean systolic and diastolic blood pressure was 126±18 mmHg and 80±11 mmHg, respectively, and median value (1Q-3Q) of creatinine was 0.8 (0.7-0.9) mg/dL (Table 2).
Inappropriate CAD screening
Only 21 patients underwent CAD screening during pre-anesthetic evaluation, corresponding to a prevalence of 2.7% (95%CI: 1.7-4.1%) of inappropriate screening for obstructive CAD in this population (Table 3). Most of these procedures were mediated by cardiology consultation, with the examination requested by the anesthesiologist in only 10% of cases (2 patients). The univariate analysis showed a potential association between CAD screening and the following variables: private hospital (P<0.048), consultation with a cardiologist (P=0.001), patients with ASA III physical status (P=0.001), and patients with the presence of stable CAD (P=0.04). In addition, patients surveyed for CAD had higher mean age (P<0.001), total body mass (P=0.003), and BMI (P=0.04).
These variables were entered into a multivariate analysis that showed the following independent predictors of inappropriate CAD screening: private hospitals (OR: 3.9; 95%CI: 1.3-11.0), patients with ASA III physical status (OR: 5.3; 95%CI: 1.7-16.2), and systemic arterial hypertension (OR: 3.8; 95%CI: 1.5-9.8) (Table 4). Screening of CAD in private hospitals totaled 3.8% (95%CI: 2.3-6.3) and in the university hospital, 1.5% (95%CI: 0.6-3.4). In ASA III patients, frequency was 9.0% (95%CI: 3.8-18.5), which was higher than the 2.1% (95%CI: 1.3-3.5) found in the other categories. Hypertensive individuals screened for CAD accounted for 5.8% patients (95%CI: 3.4-9.5), while non-hypertensive patients were only 1.3% (95%CI: 0.6-2.7).
Discussion
Randomized clinical trials have consistently demonstrated the lack of clinical effectiveness of coronary disease screening in the preoperative period of a non-cardiac surgery (20). This was a cross-sectional study to describe the prevalence of CAD screening in asymptomatic candidates for minor and medium complexity surgeries, which is considered inappropriate based on previous scientific evidence (13–16). The present study found a low prevalence of inappropriate CAD screening in asymptomatic patients who underwent pre-anesthetic consultation undergoing low-risk or intermediate non-cardiac surgeries. Screening for CAD was by far prescribed by cardiologists; only a minority of the patients was screened by the prescription of anesthesiologists. This suggests that the phenomenon of coronary disease over-diagnosis, apparently common in the preoperative period of surgery, is not mediated by pre-anesthetic evaluation.
According to the theory of behavioral economics (21,22), human behavior is based on mental shortcuts known as heuristics that lead to systematic cognitive biases that are far from a predictable and effective rational result. This reality also permeates the medical environment, and the prevailing paradigm ‘the more the better' remains on both sides of the equation, suppliers and consumers, even if the underlying motivations are from diametrically opposed spectra (22,23).
However, we did not find signs of medical overuse in the studied scenarios. Our findings were consistent with the body of evidence from the American public system in which Sheffield et al. (24) demonstrated a 3.8% prevalence of inappropriate performance of cardiac stress tests in 74,785 elective non-cardiac and non-vascular surgical patients, after surveying the Medicare database (1996 to 2008). Similarly, Kerr et al. (25) found 2.1% of cardiac tests by analyzing the Medicare database for patients undergoing minor surgeries in 2009. A review of patients who underwent cataract surgery and shoulder and knee arthroscopies in the same timeframe in the Veterans Affairs Corporate Data Warehouse have found that the prevalence of inappropriate cardiac stress tests in 22,697 veterans was also low (0.7%) (25).
In a sample of 5% of Medicare hospitalization requests between 1996 and 2008, with patients aged ≥66 years who underwent elective procedures of general surgery, urology, or orthopedics, 2,803 (3.8%) underwent preoperative cardiac stress tests without any indication (24). Our study corroborates these findings and extends this low frequency of investigation to the private environment. Although the private system has been a predictor of CAD screening with twice the requests of the public system, the absolute frequency of investigation remains low (15).
Additionally, the multivariate analysis revealed two other independent predictors of inappropriate CAD screening: patients with ASA III physical status and systemic arterial hypertension. Considering the multitude of variables related to the phenomenon, it is very strange that just three independent factors were verified. This fact probably reveals a lack of statistical power, considering we had only 21 outcomes, but it may reflect the reality, because anesthetic evaluation is strictly based on global surgical risk.
Requests for cardiovascular screening in patients without signs and symptoms of cardiovascular disease, routine assessment, and monitoring of heart disease without changes in clinical status, particularly in asymptomatic patients are considered inappropriate (26,27). Assessing the frequency of findings showing changes compatible with structural or functional heart disease in 1,071 echocardiogram exams performed at a private cardiology clinic in the countryside of the state of Bahia, Lopes et al. (28) identified that 53% of the requested examinations were inappropriate, since they would be less likely to generate beneficial effects than negative outcomes in the evaluated patients. In addition, they found that 9.4% of these tests were performed on asymptomatic patients during preoperative evaluation of a non-cardiac surgery.
A retrospective analysis of medical records of 154 patients undergoing obesity surgery at Cleveland Clinic Florida was carried out by Afolabi et al. (9), who identified that 25 (32%) of the 78 patients undergoing preoperative cardiac stress tests had positive results. All patients with positive stress test results underwent cardiac catheterization preoperatively, and cardiac angiography did not reveal significant obstructive CAD in 24 patients with abnormal findings in the study of nuclear stress. Only one patient had an obstructive lesion of the coronary artery on cardiac catheterization, requiring the placement of a coronary stent before the obesity surgery. In addition, there were five (3.2%) non-fatal cardiac events during surgery and the hospital mortality rate was 0%.
The fact that preoperative invasive procedures such as coronary artery bypass grafting or percutaneous coronary angioplasty do not reduce perioperative risk in patients with significant stable CAD is highlighted (7,10). These findings also reinforce that the request for these preoperative exams results in over-diagnosis with subsequent unnecessary treatments, without a solid evidence base that the outcome is positive considering benefits, costs, and possible damages (11,14,29).
Diagnostic overuse is commonly expected in risk mitigation scenarios and this justifies a universal tendency to screen for coronary disease. Given this finding, we must ask why anesthesiologists are immune to the inappropriate use of this diagnostic screening. First, in the pre-anesthetic consultation scenario, the primary goal of the assessment is to serve as a bridge to the surgical procedure. As the objective is surgery, an excess of diagnostic investigation would represent a barrier to the main medical action. This would also explain the low prevalence of referral for cardiology consultation. The physician's mental model in the pre-anesthetic consultation is surgical referral.
Although doctors could be affected by “attribution bias”, in this case the action is surgery and diagnostic overuse would promote surgical underuse. Therefore, viewing the surgery as an “end” would inhibit “half” of the procedures that are questionable. This is different from a routine clinical consultation when evaluation is the “end”. Thus, this study presents the following hypothesis: the prospect of a remote future procedure inhibits a futile procedure in the immediate future.
This hypothesis goes against the “bias of the present”, demonstrated by experiments in which individuals abdicate full pleasure in the future to anticipate partial pleasure. Children, for example, more commonly choose half a candy now than a whole candy in the future (30). Therefore, our second inference is that the present bias does not prevail in professional contexts, which is not exactly about pleasure itself, but the satisfaction emerging from effectively performing a professional intervention.
It is worth mentioning that the purpose of this cross-sectional study was only describing the prevalence of CAD screening; it was a study about the physician's preference, not about the patient's prognosis. We also did not perform a quantitative cost-benefit analysis for technical reasons; however, it could be evaluated in further studies. Finally, we must point out that the retrospective character of the study may be a limitation of our research, although the standardized approach of the anesthesiology teams facilitated data collection and analysis.
The present study suggested that the inappropriate screening for coronary artery disease was low in the pre-anesthetic evaluation context. Although infrequent, this practice predominantly occurs in private hospitals, with hypertensive patients, and with patients with overall poorer heath status.
References
-
1 Ministério da Saúde, Informações de Saúde, Procedimentos hospitalares do SUS. http://www2.datasus.gov.br/DATASUS/index.php?area=02 Accessed March 1, 2020.
» http://www2.datasus.gov.br/DATASUS/index.php?area=02 -
2 Thom T, Haase N, Rosamond W, Howard VJ, Rumsfeld J, Manolio T, et al. Heart disease and stroke statistics--2006 update: a report from the American Heart Association Statistics Committee and Stroke Statistics Subcommittee. Circulation 2006; 113: e85–e151, doi: 10.1161/CIRCULATIONAHA.105.171600.
» https://doi.org/10.1161/CIRCULATIONAHA.105.171600 -
3 Mohan V, Deepa R, Rani SS, Premalatha G, Chennai Urban Population Study (CUPS No.5). Prevalence of coronary artery disease and its relationship to lipids in a selected population in South India: The Chennai urban population study (CUPS No. 5). J Am Coll Cardiol 2001; 38: 682–687, doi: 10.1016/S0735-1097(01)01415-2.
» https://doi.org/10.1016/S0735-1097(01)01415-2 -
4 Grayburn PA, Hillis LD. Cardiac events in patients undergoing noncardiac surgery: shifting the paradigm from noninvasive risk stratification to therapy. Ann Intern Med 2003; 138: 506–511, doi: 10.7326/0003-4819-138-6-200303180-00017.
» https://doi.org/10.7326/0003-4819-138-6-200303180-00017 -
5 Kable AK, Gibberd RW, Spigelman AD. Adverse events in surgical patients in Australia. Int J Qual Heal Care 2002; 14: 269–276, doi: 10.1093/intqhc/14.4.269.
» https://doi.org/10.1093/intqhc/14.4.269 -
6 Gawande AA, Thomas EJ, Zinner MJ, Brennan TA. The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery 1999; 126: 66–75, doi: 10.1067/msy.1999.98664.
» https://doi.org/10.1067/msy.1999.98664 -
7 Patel AY, Eagle KA, Vaishnava P. Cardiac risk of noncardiac surgery. J Am Coll Cardiol 2015; 66: 2140–2148, doi: 10.1016/j.jacc.2015.09.026.
» https://doi.org/10.1016/j.jacc.2015.09.026 -
8 Widmer RJ, Cullen MW, Salonen BR, Sundsted KK, Raslau D, Mohabbat AB, et al. Cardiac events after noncardiac surgery in patients undergoing preoperative dobutamine stress echocardiography: findings from the Mayo Poce-DSE Investigators. Am J Med 2018; 131: 702.e15–702.e22, doi: 10.1016/j.amjmed.2017.12.025.
» https://doi.org/10.1016/j.amjmed.2017.12.025 -
9 Afolabi BA, Novaro GM, Szomstein S, Rosenthal RJ, Asher CR. Cardiovascular complications of obesity surgery in patients with increased preoperative cardiac risk. Surg Obes Relat Dis 2009; 5: 653–656, doi: 10.1016/j.soard.2009.06.009.
» https://doi.org/10.1016/j.soard.2009.06.009 -
10 Wesorick DH, Eagle KA. The preoperative cardiovascular evaluation of the intermediate-risk patient: new data, changing strategies. Am J Med 2005; 118: 1413, doi: 10.1016/j.amjmed.2005.07.068.
» https://doi.org/10.1016/j.amjmed.2005.07.068 -
11 Sampaio RO, Jonke VM, Falcão JL, Falcão S, Spina GS, Tarasoutchi F, et al. Prevalence of coronary artery disease and preoperative assessment in patients with valvopathy. Arq Bras Cardiol 2008; 91: 183–186, 200–204, doi: 10.1590/S0066-782X2008001500010.
» https://doi.org/10.1590/S0066-782X2008001500010 - 12 Maddox TM. Preoperative cardiovascular evaluation for noncardiac surgery. Mt Sinai J Med 2005; 72: 185–192.
-
13 Fleisher LA, Fleischmann KE, Auerbach AD, Barnason SA, Beckman JA, Bozkurt B, et al. 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014; 130: e278–e333, doi: 10.1161/CIR.0000000000000106.
» https://doi.org/10.1161/CIR.0000000000000106 -
14 Choosing Wisely Initiative of the ABIM Foundation. http://www.choosingwisely.org/our-mission/ Acesssed January 1, 2020.
» http://www.choosingwisely.org/our-mission/ -
15 Ferrando A, Ivaldi C, Buttiglieri A, Pagano E, Bonetto C, Arione R, et al. Guidelines for preoperative assessment: impact on clinical practice and costs. Int J Qual Heal Care. 2005; 17: 323–329, doi: 10.1093/intqhc/mzi039.
» https://doi.org/10.1093/intqhc/mzi039 - 16 National Institute for Health and Clinical Excellence (NICE). Preoperative Tests. The Use of Routine Preoperative Tests for Elective Surgery. London: National Collaborating Centre for Acute Care; 2003.
-
17 Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF, et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999; 100: 1043–1049, doi: 10.1161/01.CIR.100.10.1043.
» https://doi.org/10.1161/01.CIR.100.10.1043 -
18 Hlatky MA, Boineau RE, Higginbotham MB, Lee KL, Mark DB, Califf RM, et al. A brief self-administered questionnaire to determine functional capacity (The Duke Activity Status Index). Am J Cardiol 1989; 64: 651–654, doi: 10.1016/0002-9149(89)90496-7.
» https://doi.org/10.1016/0002-9149(89)90496-7 -
19 American Society of Anesthesiologists (ASA), ASA Physical Status Classification System. https://www.asahq.org/standards-and-guidelines/asa-physical-status-classification-system Acessed January 8, 2020.
» https://www.asahq.org/standards-and-guidelines/asa-physical-status-classification-system -
20 Kalesan B, Nicewarner H, Intwala S, Leung C, Balady GJ. Pre-operative stress testing in the evaluation of patients undergoing non-cardiac surgery: A systematic review and meta-analysis. PLoS One 2019; 14: e0219145, doi: 10.1371/journal.pone.0219145.
» https://doi.org/10.1371/journal.pone.0219145 - 21 Thaler RH. Misbehaving. New York: W.W. Norton & Company; 2016.
-
22 Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974; 185: 1124–1131, doi: 10.1126/science.185.4157.1124.
» https://doi.org/10.1126/science.185.4157.1124 -
23 Tversky A, Kahneman D. The framing of decisions and the psychology of choice. Science 1981; 211: 453–458, doi: 10.1126/science.7455683.
» https://doi.org/10.1126/science.7455683 -
24 Sheffield KM, McAdams PS, Benarroch-Gampel J, Goodwin JS, Boyd CA, Zhang D, et al. Overuse of preoperative cardiac stress testing in medicare patients undergoing elective noncardiac surgery. Ann Surg 2013; 257: 73–80, doi: 10.1097/SLA.0b013e31826bc2f4.
» https://doi.org/10.1097/SLA.0b013e31826bc2f4 -
25 Kerr EA, Chen J, Sussman JB, Klamerus ML, Nallamothu BK. Stress testing before low-risk surgery: so many recommendations, so little overuse. JAMA Intern Med 2015; 175: 645, doi: 10.1001/jamainternmed.2014.7877.
» https://doi.org/10.1001/jamainternmed.2014.7877 -
26 Bhatia RS, Alabousi M, Dudzinski DM, Weiner RB. Appropriate use criteria: a review of need, development and applications. Expert Rev Cardiovasc Ther 2016; 14: 281–290, doi: 10.1586/14779072.2016.1131125.
» https://doi.org/10.1586/14779072.2016.1131125 -
27 American College of Cardiology Foundation Appropriate Use Criteria Task Force; American Society of Echocardiography; American Heart Association; American Society of Nuclear Cardiology; Heart Failure Society of America; Heart Rhythm Society, et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 appropriate use criteria for echocardiography. J Am Coll Cardiol 2011; 57: 1126–1166, doi: 10.1016/j.jacc.2010.11.002.
» https://doi.org/10.1016/j.jacc.2010.11.002 -
28 Lopes JR, Oliveira AC, Rios VG, Correia LCL. Low prevalence of relevant findings in inappropriate echocardiograms and discordant perceptions between cardiologists and patients. Braz J Med Biol Res 2018; 51: e7413, doi: 10.1590/1414-431x20187413.
» https://doi.org/10.1590/1414-431x20187413 -
29 Peterson B, Ghahramani M, Emerich M, Foy AJ. Frequency of appropriate and low-risk noncardiac preoperative stress testing across medical specialties. Am J Cardiol 2018; 122: 744–748, doi: 10.1016/j.amjcard.2018.05.007.
» https://doi.org/10.1016/j.amjcard.2018.05.007 -
30 Zanella AV, Siqueira MJT, Lhullier LA, Molon SI. Psicologia e Práticas Sociais. Rio de Janeiro: Centro Edelstein de Pesquisas Sociais; 2008, doi: 10.7476/9788599662878.
» https://doi.org/10.7476/9788599662878
Publication Dates
-
Publication in this collection
08 Jan 2021 -
Date of issue
2021
History
-
Received
23 June 2020 -
Accepted
2 Oct 2020