Value-added measures in higher education: a historical contextualization of Brazilian experiences

Fernandes, Vivian Duarte Couto; Miranda, Gilberto José; Alexander, Nicola

doi:10.24109/2176-6681.rbep.101i259.4469

Acessibilidade / Reportar erro

Brasil

Revista Brasileira de Estudos Pedagógicos

Español English

Brasil

Español English

sumário « anterior atual seguinte »

Sumário

ESTUDOS • Rev. Bras. Estud. Pedagog. 101 (259) • Sep-Dec 2020 • https://doi.org/10.24109/2176-6681.rbep.101i259.4469 copy

Value-added measures in higher education: a historical contextualization of Brazilian experiences

Modelos de valor agregado no ensino superior: uma contextualização histórica da experiência brasileira

Modelos de valor agregado en la educación superior: una contextualización histórica de la experiencia brasileña

Authorship SCIMAGO INSTITUTIONS RANKINGS

Abstract:

School quality assessment has been the object of research of economists, educators, policymakers, and various stakeholders worldwide. In Brazil, the National Assessment System for Higher Education is an initiative that seeks to assess the country’s undergraduate programs, their faculty, and student achievement, as well as to provide quality indicators that account for the differences between them. One such indicator is the Indicator of Difference Between Observed and Expected Achievements (IDD), which measures the contribution of undergraduate programs to student achievement. Since 2006, policymakers have altered their estimation methodology, seeking to improve it as an accurate value-added measure. This research aims to discuss this change and its impacts on the ranking of undergraduate programs in Accounting in Brazil. A quantitative design was used to test four value-added models included in the historical records of the IDD and to identify the impact of these methodologies. The sample consisted of 30,668 students from 911 accounting schools. The results show that the current model is more accurate than the previous ones. However, both the literature and the findings indicate that the model could be improved by introducing explanatory variables for academic achievement that cannot be controlled by higher education institutions. The IDD is still unsuitable because it considers all institutions and all students equal.

Keywords:
higher education; quality; school effectiveness

Variable	Until 2010	2011-2012	2013 - date
Doc (Faculty members with Ph.D.s)	0.20	0.15	0.15
Me (Faculty members with master’s degrees)	0.05	0.075	0.075
RT (Full-time professors)	0.05	0.075	0.075
IFF (infrastructure)	0.05	0.075	0.05
ODP (pedagogic teaching structure)	0.05	0.075	0.075
OAF (opportunity for further training/learning)	-	-	0.025
IDD (value-added)	0.30	0.35	0.35
C (Average of seniors' Enade score)	0.15	0.20	0.20
I (Average of freshmen Enade score)	0.15	-	-

Level of analysis	Name of variable (code)	Type of variable	Value of variable
Response Variable	Enade score (C)	Numerical	0 to 100
Explanatory variable (student-related)	Mean Enem score ()	Numerical	0 to 1000
	Score in Enem’s ‘Natural Sciences’ section (CN)	Numerical	0 to 1000
	Score in Enem’s ‘Humanities’section (CH)	Numerical	0 to 1000
	Score in Enem’s ‘Mathematics’section (MT)	Numerical	0 to 1000
	Score in Enem’s ‘Languages’ section (LT)	Numerical	0 to 1000
	Ratio of parents with higher education (w)	Numerical	0 to 1
Explanatory variable (program-related)	Faculty members with Ph.Ds (Doc)	Numerical	1 to 5
	Faculty members with master’s degrees (Me)	Numerical	1 to 5
	Faculty members under full-time employment contracts (RT)	Numerical	1 to 5
	Program’s infrastructure (IFF)	Numerical	1 to 5
	Program’s pedagogical teaching structure (ODP)	Numerical	1 to 5
	Program’s opportunity for further training/learning (OAF)	Numerical	1 to 5

Program ranking	Value-added estimate compared to the sample mean	Index	Interpretation
Level 5: Most effective	At least 2 standard errors above	Higher than or equal to 2.00	Significant evidence that students exceeded the Growth Standard.
Level 4:Above-average effectiveness	Between 1 and 2 standard errors above	Between 0.99 and 1.99	Moderate evidence that students exceeded the Growth Standard.
Level 3: Average effectiveness	Between 1 standard error above and 1 standard error below	Between -1.00 and 1.00	Evidence that students met the Growth Standard.
Level 2: Approaching average effectiveness	Between 1 and 2 standard errors below	Between -1.99 and -0.99	Moderate evidence that students did not meet the Growth Standard.
Level 1: Least effective	More than 2 standard errors below	Less than or equal to -2.00	Significant evidence that students did not meet the Growth Standard.

Variable	Model 1	Model 2	Model 3	Model 4
Intercept	-32.46*	-31.92*	-32.06*	-8.09*
CN (Score in Enem’s ‘Natural Sciences’sections)	0.02*	-	-	0.02*
CH (Score in Enem’s ‘Humanities’ sections)	0.02*	-	-	0.02*
LT (Score in Enem’s ‘Languages’ sections)	0.04*	-	-	0.03*
MT (Score in Enem’s ‘Mathematics’ sections)	0.05*	-	-	0.02*
Enem score	-	0.13*	0.13*	-
W (ratio of students whose parents have higher education)	-2.76**	-2.45	-2.40	-
Doc ( faculty members with Ph.D.s)	0.02	0.06	0.06	-
Me (faculty members with master’s degrees)	0.18	0.18	0.18	-
RT (full-time professors)	0.39*	0.32*	0.32*	-
IFF (infrastructure)	0.78*	1.07*	1.17*	-
ODP (pedagogical teaching structure)	0.53**	0.31	0.37	-
OAF (opportunity for further training/learning)	-	-	-0.17	-

	Student level			Institution level
	Enade	General training	Specific training	Enade	General training	Specific training
Enem	0.54	0.37	0.51	0.71	0.61	0.68
CN	0.44	0.31	0.41	0.63	0.56	0.60
CH	0.43	0.31	0.40	0.60	0.55	0.56
LT	0.45	0.33	0.41	0.66	0.57	0.62
MT	0.44	0.26	0.43	0.69	0.53	0.67

	Model 1	Model 2	Model 3
Model 1	1.00	-	-
Model 2	0.98	1.00	-
Model 3	0.98	1.00	1.00
Model 4	0.76	0.78	0.78

Groups	Undergraduate programs	Sum	Mean	Variance
IDD_M1	911	2023.79	2.22a	0.48
IDD_M2	911	1952.97	2.14a	0.52
IDD_M3	911	1943.91	2.13a	0.51
IDD_M4	911	2152.04	2.36b	0.21

CPC (continuous)	CPC (categorical)
0 ≤ CPC < 0.945	1
0.945 ≤ CPC < 1.945	2
1.945 ≤ CPC < 2.945	3
2.945 ≤ CPC < 3.945	4
3.945 ≤ CPC ≤ 5	5

CPC levels	Model 1	Model 2	Model 3	Model 4
Level 1	4	5	5	5
Level 2	145	160	161	101
Level 3	593	586	586	639
Level 4	165	156	155	161
Level 5	4	4	4	5

Effectiveness level	Model 1	Model 2	Model 3	Model 4
Level 1 - Least effective	14	15	16	78
Level 2 - Above average effectiveness	68	59	57	110
Level 3 - Average effectiveness	578	587	587	532
Level 4 - Above average effectiveness	192	189	189	102
Level 5 - Most effective	59	61	62	89

	2-level down-grade	2-levels upgrade	t test (p-value)	Interpretation (The most effective programs have…)
Enade mean	42.71	50.96	0.00	Highest overall score on the Enade
Mean Enade score for FG	52.91	58.53	0.00	Highest Enade score for FG
Mean Enade score for CE	39.28	48.41	0.00	Highest Enade score for CE
Mean books read/year	0.43	0.55	0.05	Highest % of people eading 3 books or +
Mean hours of study	0.28	0.51	0.00	Highest % of people studying above 3h
Mean Enem score for CN	490.83	501.31	0.05	Highest Enem score for Natural Science
Mean Enem score for CN	509.80	540.18	0.00	Highest Enem score for History
Mean Enem score for LT	533.45	546.44	0.00	Highest Enem score for Literature
Mean Enem score for LT	556.93	574.22	0.00	Highest Enem score for Math
Mean IGC (continuous)	2.52	3.00	0.00	Highest general Index
Mean ODP	2.26	4.29	0.00	Highest pedagogical teaching
Mean IFF	2.27	4.48	0.00	Highest infrastructure
Mean OAF	2.26	4.20	0.00	Highest opportunity for further training/learning
Mean Me	2.87	3.99	0.00	Highest % of faculty members with masters degrees
Mean Doc	1.16	1.84	0.04	Highest % of faculty members with Ph.D.s

		IDD_M1	IDD_M2	IDD_M3	IDD_M4
Marital status	ρpb	0.20*	0.20*	0.20*	0.04*
Ethnicity	ρpb	0.05*	0.06*	0.06*	0.04*
Nationality	ρpb	0.02*	0.02*	0.02*	0.01*
Level of education (father’s)	ρpb	-0.02*	-0.02*	-0.01*	0.00
Level of education (mother’s)	ρpb	-0.01*	-0.01*	-0.01	0.00
Income	ρpb	0.06*	0.06*	0.06*	0.03*
Work	ρpb	-0.00	0.00	0.00	0.02*
Funding	ρpb	-0.17*	-0.19*	-0.19*	0.01
Affirmative policy	ρpb	-0.04*	-0.04*	-0.04*	-0.00
High school institution	ρpb	0.01	0.01*	0.01*	0.01
Type of high school	ρpb	0.12*	0.12*	0.12*	0.02*
First in the family in a HEI	ρpb	-0.02*	-0.02*	-0.02*	-0.01
No. of books read	ρpb	-0.02*	-0.02*	-0.02*	-0.02*
Hours of study	ρpb	-0.05*	-0.04*	-0.05*	-0.05*
Enem (CN - Natural Sciences)	ρ	0.03*	0.02*	0.02*	0.04*
Enem (CH - Humanities)	ρ	0.03*	0.01	0.01	0.03*
Enem (LT - Languages)	ρ	0.04*	0.04*	0.04*	0.04*
Enem (MT - Mathematics)	ρ	0.03*	0.05*	0.04*	0.06*

		IDD_M1	IDD_M2	IDD_M3	IDD_M4
Academic organization	ρpb	0.02	0.02	0.02	0.11*
No. of programs in the HEI	ρ	0.14*	0.16*	0.16*	0.24*
IGC (continuous)	ρ	0.35*	0.34*	0.34*	0.42*
Learning modality	ρpb	0.11*	0.04	0.04	0.05
ODP	ρ	0.35*	0.35*	0.35*	0.26*
IFF	ρ	0.36*	0.38*	0.38*	0.31*
OAF	ρ	0.35*	0.36*	0.36*	0.30*
Faculty members with master’s degrees	ρ	0.12*	0.13*	0.13*	0.18*
Faculty members with Ph.D.s	ρ	0.07*	0.08*	0.07*	0.11*
Employment contract	ρ	0.15*	0.13*	0.13*	0.16*

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira Setor de Indústrias Gráficas - Quadra 04 - lote 327, Térreo, Ala B, CEP 70.610-440 Brasília-DF Brasil, Telefones: (61) 2022-3077, 2022-3078 - Brasília - DF - Brazil
E-mail: editoria.rbep@inep.gov.br

Acompanhe os números deste periódico no seu leitor de RSS

[TFN1] Source: Elaborated by the authors based on Brasil. Inep (2009aBRASIL. Lei nº 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior - Sinaes e dá outras providências. Diário Oficial da União, Brasília, DF, 15 abr. 2004. Seção 1, p. 3.), Brasil. Inep. (2012BRASIL. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Inep). Diretoria de Avaliação da Educação Superior (DAES). Nota Técnica nº 29, de 15 de outubro de 2012. Cálculo do Conceito Preliminar de Curso (CPC) referente ao ano de 2011. Brasília, DF: Inep , 2012. Disponível em: <Disponível em: http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2011/nota_tecnica_indicadores_2011_2.pdf >. Acesso em: 20 nov. 2020.
http://download.inep.gov.br/educacao_sup... ); Brasil. Inep. (2014aBRASIL. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Inep). Nota Técnica nº 72. Cálculo do conceito preliminar de curso referente a 2013. Brasília, DF: Inep , 2014a. Disponível em: <Disponível em: http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2013/nota_tecnica_n_72_2014_calculo_cpc_2013.pdf >. Acesso em: 27 nov. 2020.
http://download.inep.gov.br/educacao_sup... ).