Resources, innovation and performance in labor courts in Brazil

Recursos, innovación y desempeño en los tribunales laborales en Brasil

Abstract

This article examines the relationships between resources, innovation, and performance in courts. Data from 24 Brazilian labor courts covering the period between 2003 and 2013 were used to develop theoretical/empirical models using Data Envelopment Analysis and stochastic frontier analysis. The results indicate that there was improvement in the performance of the courts during the period examined. This improvement owed more to the adoption of innovations than to variation in technical efficiency. Critical periods for the adoption of the electronic judicial process (in 2006 and 2012) had a negative impact on the index related to innovation adoption. The stochastic model shows that court size and investment in the training of personnel are key factors for explaining the variation in the efficiency of the courts.

Keywords:
innovation; performance; judicial administration

Resumen

El presente estudio examina las relaciones entre los recursos, la innovación y el desempeño en los tribunales. Se utilizaron datos de 24 tribunales laborales brasileños que cubrían el período comprendido entre 2003 y 2013 para desarrollar modelos teóricos/empíricos utilizando el análisis por envoltura de datos y análisis de frontera estocástica. Los resultados indican que hubo mejoras en el desempeño de los tribunales durante el período examinado. Esta mejora se debe más a la adopción de innovaciones que a la variación en la eficiencia técnica. Los períodos críticos para la adopción del proceso judicial electrónico (en 2006 y 2012) tuvieron un impacto negativo en el índice relacionado con la adopción de la innovación. El modelo estocástico muestra que el tamaño de la corte y la inversión en la formación del personal son factores clave para explicar la variación en la eficiencia de los tribunales.

Palabras clave:
innovación; desempeño; administración judicial

Resumo

O presente estudo examina as relações entre recursos, inovação e desempenho em tribunais. Foram utilizados dados de 24 tribunais trabalhistas brasileiros no período entre 2003 e 2013. Foram desenvolvidos modelos teóricos/empíricos utilizando a análise envoltória de dados e a análise de fronteira estocástica. Os resultados indicam que houve melhora no desempenho dos tribunais durante o período estudado. Essa melhoria deve-se mais em função da adoção de inovações do que da variação da eficiência técnica. Os períodos críticos para a adoção do processo judicial eletrônico (em 2006 e 2012) tiveram um impacto negativo no índice relacionado com a adoção de inovação. O modelo estocástico indicou que o tamanho do tribunal e o investimento na formação de pessoal foram fatores-chave para explicar a variação na eficiência dos tribunais.

Palavras-chave:
inovação; desempenho; administração judicial

1. Introduction

In the context of reforms of public administration, pressure has been put on public organizations to improve efficiency, by reducing budgets, raising expectations about the quality of services provided, and introducing performance-based management techniques (Boyne et al., 2005; Osborne and Gaebler, 1993). Assessment of performance of public organizations is now a key aspect in public administration (Boyne, 2004). In recent decades, procedure and technical innovations have been adopted in Brazilian courts. The most important of these innovations are those supported by information and communication technologies (ICTs) and, more specifically, the electronic judicial process. It is important to understand the extent of the effects caused by and the resources involved in the adoption of an innovation by the judiciary, as exemplified by the electronic judicial process, because it is a subject that has been under-explored (Sousa and Guimaraes, 2014).

The objectives of this paper are: (a) to develop and test a theoretical and empirical model that explains the relationships between resources, innovation, and performance in courts and (b) to measure the observed performance (variation in efficiency and adoption of innovations) of 24 Brazilian labor courts as a function of the resources and level of innovation in those courts. The study examines the attributes that may have an impact in the performance of Brazilian labour courts and thus contributes to knowlegde about judicial administration, providing evidence for improvement of the management of courts and the development of better policies for the judiciary. The judiciary attached to labor courts was chosen because it has the following relevant characteristics: shorter length of the judicial process (Dias Júnior, 2004) and; high level of innovation adoption (Costa, 2008).

The Brazilian labor courts are comprised of: (i) first trial courts, or courts in the first-degree, where a judge sitting alone decides the outcome of a case; (ii) Regional Labor Courts, or courts in the second-degree; and (iii) the Superior Labor Court, or the court of final appeal. The judicial process in courts in the first-degree begin when an application is assigned to a judge. The first stage requires an attempt at conciliation. If conciliation is successful, the judge records the conclusion. If conciliation is unsuccessful, the judge decides the case. Courts in the second-degree courts can judge new applications or may receive appeals against decisions of the courts in the first-degree. The new cases relate to specific issues that arise in situations such as collective bargaining among others. New cases may be decided by a judge sitting alone, while appeals are reviewed by a panel of judges that constitute the appeal court if one party appeals a decision. Rulings of the courts in the second-degree may be appealed, in which case they are referred to the Superior Labor Court.

There are in Brazil 1,568 courts in the first-degree and 24 Regional Labor Courts (TST 2015). 32.6% of lawsuits are presented in electronic format. There are 3,371 labor judges, 36,997 permanent officers and administrators of the courts. In 2013 the court system handled 7.9 million lawsuits, 3.9 million pending cases and 4 million new cases (CNJ, 2014).

2. Theoretical background and hypotheses

The literature on the performance of the judiciary identifies several variables that contribute to the provision of legal services. The production frontiers approach has been developed using the technique called Data Envelopment Analysis (DEA) in several studies that evaluate the efficiency of courts (Deyneli, 2012; Kittelsen and Førsund, 1992; Lewin, Morey and Cook, 1982; Pedraja-Chaparro and Salinas-Jimenez, 1996; Schneider, 2005; Yeung and Azevedo, 2011). The assessment of efficiency through stochastic frontier analysis, a different technique that was used by Castro (2011) and Schwengber (2006), is less commonly used.

The DEA technique allows the assessment of performance through relative composite measures of efficiency, and does not require previous knowledge of prices, costs, and relationships among the variables. In addition, it makes it possible to assess one organizational decision-making unit (DMU) of analysis in comparison to other units to identify the most productive ones. These most productive units are then used as references, also described as “best practice” (Zhu, 2009). In this study, each court is regarded as a decision-making unit (DMU).

The index proposed by Malmquist (1953) makes it possible to investigate the variation in technical efficiency and technological progress in a dynamic manner over time. It also makes it possible to decompose the multiplicative effect on the total productivity variation index into separate components of technical efficiency and innovation adoption - technological change (Cooper, Seiford and Tone, 2007). The variation in efficiency, also known as catch-up, is measured by the distance between the index of a given court and the production frontier. Thus, catch-up represents the extent to which the efficiency of the court has improved or deteriorated over time (Cooper, Seiford and Tone, 2007). Technological change (known as frontier-shift), which is a result of the adoption of innovations, is assessed by the displacement of the optimal frontier of pooled resources, and is measured by the geometric mean of change between two periods (Färe et al., 1994). This index was used in research on the judiciary by Kittelsen and Førsund (1992). The advantage of using this index is that the impact of the adoption of innovation can be evaluated in a longitudinal approach.

Turning to the other possible approach, stochastic frontier analysis requires the development of a more rigorous econometric model to explain the extent of variation in judicial performance by the variation in the productive resources of the judiciary. This technique makes it possible to assess the factors that affect the efficiency and the decomposition of the error, to identify systematic effects on efficiency (Castro, 2011). This technique also makes it possible to compare performance with operations that are a reference. So, we decided to take advantage of the strengths from both approaches, using Malmquist index, a non-parametric modeling, and stochastic frontier, a parametric one.

Box 1 presents a non-exhaustive summary of the variables found in the literature that are employed in the model used in this study. The first column contains indicators that exhibit dependency relationships in parametric models, in addition to the resource or result in non-parametric models. The second column shows the variables. The third column lists the studies that support the findings that are listed.

Box 1
Main variables from studies on judicial performance

The basic quantitative model used in these studies, increased or reduced, mainly depending on the availability of data, has the relationship presented in equation 1.

$y=f(Pending,new,Magistrate,Staff)$ (1)

Where Y, the dependent variable, is understood as completed/resolved cases. The independent variables are Pending, pending cases; New, new cases assigned to judges; Magistrate, the number of judges; and Staff, the number of support staff. The variables Pending and New are related to the workload of the judge.

The application of Information and Communication Technologies (ICT) by the judiciary have been studied both in relation to the effect on performance and terms of the adoption of innovation (Sousa and Guimaraes, 2014). The recurring themes are the impact of these technologies on judicial performance (Crunkilton, 2009; Joia, 2008, 2009; McKechnie, 2003; Rosa, Teixeira and Pinto, 2013; Saman and Haider, 2013; Velicogna, Errera and Derlange, 2011, 2013), e-government (Joia, 2008, 2009; McKechnie, 2003), systems and databases (Hara, 2007; Rosa, Teixeira and Pinto, 2013), and the electronic judicial process (Velicogna, Errera and Derlange, 2011, 2013).

Studies that address performance in the judiciary can contribute to the evaluation of innovation through the measurement of parametric and non-parametric relationships among the main resources of the judiciary and their performance. Studies that aim to evaluate the effect of innovation on judicial outcomes use measures that have already been tested and validated in different contexts (Sousa and Guimaraes, 2014). The pressure caused by the increase in judges’ workload has been shown to have an impact on the performance of courts (Beenstock and Haitovsky, 2004; Dimitrova-Grajzl et al., 2012; Lewin, Morey and Cook, 1982; Rosales-López, 2008), so that the following hypothesis can be formulated:

H₁: The workload in labor courts has a positive impact on performance.

The organization size variable is traditionally used in innovation adoption studies (Boyne et al., 2005; Damanpour and Schneider, 2009; Damanpour and Wischnevsky, 2006; Rogers, 2003). As suggested by Rogers (2003), this variable would be positively related to innovation adoption, and therefore, large courts would be more likely to adopt an innovation. On the performance perspective is expected that large organizations would be more efficient (Tulkens, 1993), in this way, large courts would increase performance (Rosales-López, 2008). Thus, the following hypothesis is suggested:

H₂: Size positively impacts court performance.

To assess innovation in labor courts, the variable electronic process index (II) was used as an indicator and represents the percentage of electronic processes adopted by the labor judiciary, together with, as a proxy measure, total investment in ICT. These variables for measuring innovation in the judiciary have severe limitations, but it is difficult to identify indicators that assess the indirect and final impacts, as has previously been noted by Rauen (2013).

The variable investment in ICT represents the share of resources invested by the court in activities involving the electronic process, and the electronic process index is related to the ability of the court to operationalize innovation. There is no consensus on the relationship between ICT investment and organizational performance. The paradox, first identified by Solow (1987) is that there is a negative relationship between these variables, even though there are studies that indicate that such investments may contribute to improve performance in organizations (Stratopoulos and Dehning, 2000). In the public sector, investment in ICT can improve the efficiency of the services provided (Foley and Alfonso, 2009; Lee and Perry, 2002). Nevertheless in the judiciary there is evidence that this investment has both positive (Soares and Sviatschi, 2010) and negative (Buscaglia and Ulen, 1997; Deyneli, 2012) impacts on court performance. Thus hypotheses 3 and 4 are proposed as follows:

H₃: The judicial electronic process index positively affects court performance.

H₄: Investment in ICT positively affects court performance.

Investment in training in courts can be an important resource by providing capability related to legal and managerial knowledge. Training can be a stage of the ICT implementation process (Crunkilton, 2009; Rosa, Teixeira and Pinto, 2013; Velicogna, Errera and Derlange, 2011, 2013). In addition, the adoption of technology without training may limit its impact on performance (Aragão, 1997; Buscaglia and Ulen, 1997; McKechnie, 2003; Velicogna, 2007). Yeung and Azevedo (2011) have found evidence that judicial administration - management training is one of the variables - is positively correlated with performance. As a result, the following hypothesis is proposed:

H₅: Investment in training in courts has a positive relationship with performance.

3. Method

The labor judiciary in Brazil is composed of 24 courts divided into jurisdictions: 1. Rio de Janeiro; 2. São Paulo (Capital), covering the metropolitan area of São Paulo, part of the metropolitan area of the Baixada Santista region and Ibiúna city; 3. Minas Gerais; 4. Rio Grande do Sul; 5. Bahia; 6. Pernambuco; 7. Ceará; 8. Pará and Amapá; 9. Paraná; 10. Federal District and Tocantins; 11. Roraima and Amazonas; 12. Santa Catarina; 13. Paraíba; 14. Acre and Rondônia; 15. São Paulo (Campinas) cities included in the 2^nd jurisdiction; 16. Maranhão; 17. Espírito Santo; 18. Goiás; 19. Alagoas; 20. Sergipe; 21. Rio Grande do Norte; 22. Piauí; 23. Mato Grosso; and 24. Mato Grosso do Sul.

This study uses census data, covering all 24 labor court jurisdictions for courts in both the first and second-degree, and draws on secondary data available in the Justice in numbers database of the Brazilian National Council of Justice (CNJ) and data collected from the websites of the courts for the period between 2003 and 2013. It is important to notice that this Database evolved overtime, improving the collecting data method and including more variables and analysis.

The analyses were performed in two steps: 1) construction, analysis, and processing of the database and 2) performance optimization modelling and constructing a classification of courts using the Malmquist index based on DEA and econometric modelling using stochastic frontier analysis.

The collected data were structured into a two-dimensional database. First, these data were analyzed regarding measurement units, transformations, missing values, and outliers. Next, descriptive statistics and a correlation matrix were used for data and model analysis. Finally, the model parameters were estimated. The Deap Version 2.1 software was used for the estimation of the DEA results, and the Frontier Version 4.1 software was used for the estimation of the stochastic frontier parameters. The R software was used for the analysis of the secondary data, descriptive statistics, and correlation analysis.

The courts were classified according to technical efficiency and innovation adoption. For the analysis of the relationships between resources, innovation, and performance, panel data models of Malmquist indices were developed using DEA programming (nonparametric), and the variable parameters were estimated using stochastic frontier analysis (parametric). Data from the CNJ made it possible to break down some variables and distinguish between courts in the first and second-degrees. Therefore, the data were evaluated in these two degrees, at both the chamber level and court level. Box 2 shows the variables used in the study.

Box 2
Variables used in the study

The variables Staff, ICT, II, IE, and Size enrich the analysis of the resources and capabilities that impact performance in the judiciary and may be useful in the evaluation of inefficiency. The variable IE is not available in the Justice in numbers database of the CNJ and was collected from the website of each court. The variables Staff and ICT are not separated into first and second-degree. The variable II is called the electronic process index by the CNJ and was first published in 2009.

The variables related to resources and internal capabilities of the previous model were used as the resources (inputs) - variables Magistrate and Staff - and the variable Y was used as the result (output), divided between courts in the first and second-degree. These variables are already established in the literature, and it was not possible to add other variables in the Malmquist model as the data was missing. The model follows the suggestion of Charnes, Cooper and Rhodes (1978), which assumes constant returns to scale with product orientation, already applied in the public sector, and which is also one of the models used in the literature on performance in the judiciary. This approach is appropriate for the Brazilian context, given that, according to the civil law characteristics adopted in Brazil, decisions tend to conform to codes that are different from common law, where the emphasis is on jurisprudence (Deyneli, 2012; Yeung and Azevedo, 2011). The orientation toward product - maximizing product while maintaining the resource base - is justified because, in addition to having been widely adopted in the literature, there is little flexibility to change the resource base of courts, for example by dismissing judges or support staff (Yeung and Azevedo, 2011).

Resources, capabilities, and results were divided by the judge’s workload. The weighting by the workload, according to Yeung and Azevedo (2011), is critical for controlling the variation between courts. The workload indicator, as provided by the CNJ (2013) in the Justice in numbers database, is understood in Brazil as follows:

${Workload}_{it}=\frac{{New}_{it}+{Pending}_{it}+{IR}_{it}+{PR}_{it}}{M_{it}}$ (2)

Where Workload is the average workload per judge in each court and period, IR is the amount of internal resources, and PR is the amount of pending internal resources, per given court and period. This definition of workload is more complete in that it includes legal resources that were ignored by Beenstock and Haitovsky (2004), and without which the workload may incur measurement problems. In addition, it has the advantage of being more parsimonious than the definition proposed by Costa and collaborators (2006). The terms New, Pending, and Magistrate were defined previously. The stochastic frontier is based on the Cobb-Douglas function, which is also applied to the judiciary by Castro (2011) and Schwengber (2006). The stochastic frontier model is stated on Equations 3 and 4.

$\frac{ln{Y}_{it}}{{Magistrate}_{it}}={\beta }_0+{\beta }_1{lnMagistrate}_{it}+\frac{{\beta }_2{lnStaff}_{it}}{{Magistrate}_{it}}+\frac{{\beta }_3{lnNew}_{it}}{{Magistrate}_{it}}+\frac{{\beta }_4{lnPending}_{it-1}}{{Magistrate}_{it}}+Time+v_{iy}+u_{it}$ (3)

Where i = 1, 2, 3, ... 24 representing the units of analysis, i.e., the 24 Brazilian regional labour courts, and t represents the time period; vit is a random variable that represents the error and is independent of uit . The term uit refers the inefficiency effects model, i.e. a random variable with normal truncated distribution. The variable Time is a vector representing dummies for the years investigated.

$u_{it}={\delta }_0+{\delta }_1{lnICT}_{it}+{\delta }_2{Workload}_{it}+{\delta }_3{lnIE}_{it}+{\delta }_4{II}_{it}+{\delta }_{5}Size+w_{it}$ (4)

To test the model and calculate the maximum likelihood we used: (a) the suggestion of Battese and Coelli (1995) to replace σ_u ² and σ_v ² with σ²= σ_v ²+σ_u ² and γ= σ_u ²/ σ², where γ has a value between 0 and 1; and (b) the likelihood ratio test on γ to verify the adequacy of the model.

4. Results and discussion

There are some variables for which data is not available for the entire period of this research, from 2003 to 2013. The Staff1 and Staff2 variables are available only after 2009. Therefore, 144 missing values were computed, and the variable Staff was analyzed as the sum of Staff1 and Staff2. The following missing data were identified: (a) from 2003 to 2006, there are no data available for the variables II and Size; (b) from 2007 to 2008, there are no data available for the variable II; and (c) for the other years, see box 3.

Box 3
Missing data

Table 1 shows the descriptive data for the variables used in this study. The data exhibit high heterogeneity among courts. The court with the largest number of magistrates (422) has nearly 20 times the number of the smallest (22), which justifies the use of Spearman’s correlation (non-parametric), the adjustment of the variables for workload and the number of magistrates, and the transformation by natural logarithm. A high correlation was found between the following variables: (a) Magistrate with Staff, ICT, Pending, and New; (b) Staff with ICT, Pending, and New; and (c) Pending and New. With the division by the workload and the number of magistrates, the correlation decreased, remaining high only between Magistrate and Staff. The use of both variables in the model is justified because it is a specification widely used in the literature. Adjustments were also important to reduce the extreme values found.

The difference in the number of new cases between the court receiving the lowest number (10,502) and the court receiving the highest number (778,679) is more than 74 times. The average number of cases resolved in the courts in the first-degree is more than four times the number in the courts in the second-degree. The workload of the courts in the first-degree is approximately 1.5 times greater than that in courts in the second-degree. In relation to the innovation index (II), whereas some courts have almost all processes in electronic format, others have an index close to zero. The investment in ICT seems to be the most heterogeneous variable, given that the difference between the court that invested the most in one year in the analyzed period (R$ 883.395.206,00) and the court that invested the least (R$ 131,373,00) is more than 6,000 times.

Table 1
Spearman correlation and descriptive statistics

	Y2	Y1	Magistrate	Staff	ICT	Workload	Workload2	Workload1	II	II2	II1	IE	Pending	New
Y2	1.000
Y1	.902***	1.000
Magistrate	.929***	.933***	1.000
Staff	.893***	.913***	.951***	1.000
ICT	.566***	.567***	.590***	.590***	1.000
Workload	.462***	.530***	.344***	.398***	.380***	1.000
Workload2	.782***	.641***	.642***	.609***	.470***	.540***	1.000
Workload1	.332***	.424***	.217***	.279***	.298***	.971***	.373***	1,000
II	-.007	-.096	-.022	.000	.100	-.060	.117	-.066	1.000
II2	-.069	-.143	-.056	-.027	.092	-.057	.125	-.069	.875***	1.000
II1	.015	-.080	-.008	.025	.111	-.036	.144	-.038	.999***	.865***	1.000
IE	.323***	.345***	.357***	.381***	.485***	.105	.188**	.025	.364***	.370***	.375***	1.000
Pending	.840***	.853***	.808***	.816***	.560***	.747***	.690***	.642***	-.087	-.077	-.074	.279***	1.000
New	.925***	.983***	.939***	.914***	.574***	.543***	.670***	.433***	-.063	-.125	-.046	.339***	.858***	1.000
Obs	264	264	264	264	262	264	264	262	115	114	108	156	264	264
Aus	0	0	0	0	2	0	0	2	149	150	156	108	0	0
Mean	25791.92	116058.70	125.261	1372.72	R$ 33700000,00	2026.368	1474.923	2165.27	0.163	0.088	0.191	514616.80	129148.8	145617.9
SD	29962.69	125061.60	100.840	1099391	R$ 115000000,00	630.102	674.533	684.474	0.277	0.229	0.312	492819.30	154567.2	153011.8
Min	758	4594	22	225	R$ 131373,00	7.676.964	277	850	0	0	0	0	12966	10502
Max	130007	632574	422	5533	R$ 883395206,00	4054.775	3411.192	4388	1.000	1	1	2812157	797987	778679

• Source: Research data. Obs.: number of valid observations; Aus: missing data; SD: standard deviation. *** p≤ 0.01. ** p≤ 0.05

4.1 Malmquist indices

The Malmquist index, calculated here using DEA, is split between Vef - technical efficiency variation, known as catch-up - and Vtec, known as frontier-shift, being the latter associated with the adoption of innovations. Because the DEA technique is sensitive to the effect of missing data, we decided to use the resource variables Magistrate and Staff and the sum of variables Y1 and Y2 as performance variables because the other variables have missing data. The court of the 12^th region (Santa Catarina) was removed from the analysis due to missing values. These variables have been widely used in the literature on this technique. The sample comprised 253 observations. Tables 2 and 3 show the Malmquist indices for each court (DMU) and for each year.

Table 2
Performance index per court

Court	General model			Court Size
	Vef	Vtec	Malmquist
1	0.972	1.029	1.001	L
2	1.000	1.008	1.008	L
3	1.006	1.017	1.022	L
4	1.002	1.005	1.007	L
5	0.996	1.003	0.999	M
6	1.008	0.993	1.001	M
7	0.975	1.028	1.002	M
8	0.975	0.992	0.967	M
9	0.988	1.005	0.993	M
10	0.986	1.001	0.987	M
11	0.977	1.032	1.008	M
13	1.000	1.001	1.001	M
14	0.997	1.018	1.015	S
15	1.031	1.009	1.040	L
16	1.000	0.999	0.999	S
17	1.015	0.983	0.997	S
18	1.017	1.029	1.047	M
19	0.963	1.000	0.963	S
20	1.051	1.016	1.069	S
21	1.003	0.998	1.001	S
22	1.027	0.997	1.024	S
23	1.017	0.973	0.990	S
24	0.980	0.969	0.950	S
Geometric mean	0.999	1.004	1.004

• Source: Research data. Notes: Vef: technical efficiency; Vtec: Technological change; L: Large; M: Medium; S: Small; Malmquist: total productivity index.

The results indicate improvement in performance due to innovation adoption in 16 courts. These are the courts that are more innovative because they contributed to shifting the technological frontier. The other seven courts failed to contribute to performance improvement. For the period analyzed, labor courts improved performance more due to innovation adoption than because of variation in technical efficiency, although the latter index was close to 1.0. A value of 1.0 means that the court neither increased nor decreased technical efficiency. Six courts achieved improvements in technical efficiency indices and in innovation adoption.

All large courts had technological variation indices above 1.0 during the period studied, indicating that there was improvement in this index. On average, the labor courts improved performance in the Malmquist total productivity index, the result of which was 1.004. This improvement in performance was more a result of innovation adoption (1.004) than of improvement in technical efficiency (0.999).

Table 3
Performance index per year

Year	Vef	∆% Vef	Vtec	∆% Vtec	Malmquist	∆% Malmquist
2003/2004	0.996	-	1.092	-	1.088
2004/2005	0.951	-4.52%	1.052	-3.66%	1.001	-8.00%
2005/2006	1.033	8.62%	0.928	-11.79%	0.958	-4.30%
2006/2007	0.991	-4.07%	1.023	10.24%	1.014	5.85%
2007/2008	1.057	6.66%	0.988	-3.42%	1.044	2.96%
2008/2009	1.052	-0.47%	0.872	-11.74%	0.917	-12.16%
2009/2010	0.984	-6.46%	1.058	21.33%	1.041	13.52%
2010/2011	0.952	-3.25%	1.029	-2.74%	0.980	-5.86%
2011/2012	1.036	8.82%	0.936	-9.04%	0.969	-1.12%
2012/2013	0.948	-8.49%	1.092	16.67%	1.036	6.91%
Geometric mean	0.999		1.004		1.004

• Source: Research data. Notes: Δ% Vef: the percentage change in technical efficiency between the years. Δ% Vtec: percentage change of the technological change index between the years. Δ% Malmquist: variation of the Malmquist index between the years.

In 2006, Law No. 11.419 formalized the use of the electronic judicial process in Brazil. In that year, compared to 2005, the Vtec index declined, possibly due to changes and adjustments courts made to adopt this innovation. By 2007, compared to 2006, this result had already changed for the better. In 2012, the Brazilian Superior Council of Labor Justice (Conselho Superior da Justiça do Trabalho - CSJT) mandated the adoption of a unified electronic process for the labor courts. Indices from 2012, compared to 2011, exhibited a reduction in the Vtec index, also indicating that the courts may take a certain amount of time to assimilate the new technology, for example, through actions such as training and adaptation of the technology. However, in 2013, compared to 2012, an improvement in the index was observed, indicating that the system had become institutionalized. These results are in line with the perception of managers and magistrates in labor courts that the introduction of the electronic process initially led to a decrease in performance (Sousa and Guimaraes, 2017).

Table 4 shows the correlation between the performance indices and the variables used in the study.

Table 4
Spearman correlation between the performance indicators and resource variables

		II	Workload	Magistrate	Staff	ICT	IE	Pending	New
Vef		-.022	.019	-.013	-.026	.006	-.043	-.011	.017
	N	110	230	230	230	228	149	230	230
Vtec		.225**	.069	.013	.044	-.211***	.226***	.040	.033
	N	110	230	230	230	228	149	230	230
Malm		.129	.087	-.043	-.021	-.147**	-.010	-.003	.027
	N	110	230	230	230	228	149	230	230

• Source: Research data. Notes: Vef: technical efficiency. Vtec: technological change. Malmquist: total productivity index. *** p≤ 0.01. ** p≤ 0.05.

There was a positive and significant correlation between the technological variation index (Vtec) and the electronic process index (II), as was expected. The highest positive correlation occurs between the Vtec index and investment in training (IE), also an expected result, confirming results found by Sousa and Guimaraes (2017) who argue that investment in training is important in the process of adopting innovations by the labor judiciary. The variable investment in ICT is negatively correlated with Vtec and Malmquist, which was not expected, given that such investment should have the effect of improving performance.

4.2 Inefficiency model

To test the inefficiency, four models were developed, where availability of data permitted, following the specification of equations 3 and 4. Model 1 covers the entire period of the research, 2003 to 2013, and the other models cover the period 2009 to 2013. Table 5 shows the estimated models and tests. The use of double log allows the direct verification of elasticity.

Table 5
Stochastic frontier with conditional mean of inefficiency

	Model 1	Model 2	Model 3	Model 4
β 0	0.636 (2.5)***	5.673 (6.3)***	7.912 (5.912)***	9.094 (8.872)***
lnMagistrate	0.040 (3.25)***	0.130 (5.007)***	0.092 (2.21)**	0.042 (1.675)**
lnStaff/Magistrate	0.078 (1.893)**	0.161 (1.323)*	0.019 (0,265)	-0,001 (-0,10)
lnNew/Magistrate	0,785 (20,907)***	0,207 (2,726)***	0,132 (1,608)*	0,089 (1,106)
lnPending/Magistrate	0,057 (2,791)***	-0,155 (-1,696)**	-0,327 (-3,797)***	-0,379 (-4,658)***
Time2 (2004)	0,189 (5,127)***
Time3 (2005)	0,113 (2,909)***
Time4 (2006)	0,150 (3,651)***
Time5 (2007)	0,167 (4,509)***
Time6 (2008)	0,194 (5,007)***
Time7 (2009)	0.148 (4.092)***
Time8 (2010)	0.186 (4.946)***	0.031 (0.590)	0.058 (0.608)	0.056 (1.223)
Time9 (2011)	0.206 (5.259)***	0.042 (0.738)	0.058 (0.42)	0.052 (1.053)
Time10 (2012)	0.180 (4.848)***	0.044 (798)	0.087 (3.324)***	0.069 (1.353)*
Time11 (2013)	0.482 (11.599)***	0.054 (0.956)	0.097 (0.747)	0.087 (1.706)**
Inefficiency Model
δ0	-6.600 (-3.027)***	0.185 (0.185)	1.217 (4.304)***	1.616 (2.703)***
Large	-2.069 (-34)***	-0.113 (-0.881)	-0.027 (-1.288)*	-0.086 (-0.907)
Small	0.404 (3.277)***	-0.074 (-0.708)	-0.044 (-0.778)	0.018 (0.306)
LnICT	0.146 (3.698)***	0.067 (1.178)	0.005 (0.381)	0.013 (0.338)
Workload	0.001 (2.99)***	0.000 (-4.176)***	0.000 (-2.244)**	-0.001 (-5.606)***
lnIE		-0.010 (-1.297)*	-0.009 (-1.46)*	-0.010 (-1.895)**
II				0.025 (0.441)
N_adopt			-0.041 (-0.414)
Adopt			0.013 (0.220)
σ2	0.304 (2.809)***	0.022 (4.116)***	0.017 (11.618)*	0.015 (6.734)***
γ	0.982 (131.785)***	0.353 (2.1)**	0.000 (0.036)	0.181 (1.839)**
Courts	24	24	24	24
Observations	262	113	113	113
LR Test	35.970	28.280	46.370	55.450
Restrictions	6	7	9	8
Critical values (Kodde and Palm. 1986)	21.67	23.55	27.13	25.37
Log likelihood	162.767	59.239	68.286	72.825

• Source: Research data. Notes: t-statistic: * p≤ 0.1; ** p≤ 0.05; *** p≤ 0.01. The critical values of Kodde and Palm are regarding α=0,001. The t-statistic values are in parentheses.

All parameters in model 1 are statistically significant. For models 2, 3, and 4 it is possible to add certain more recently available variables, but some variables are not significant. An increase in the number of judges contributes to an increase in performance of the labor courts. This result is consistent with the study by Schwengber (2006) of the State Judiciary of Rio Grande do Sul, Brazil. However, it differs from the results found by Castro (2011) and Dimitrova-Grajzl et al. (2012) who studied, respectively, the Brazilian state courts and the Slovenian local and district courts. The number of staff and administrators contributes to the increase in performance of the labor judiciary, a result also found by Rosales-López (2008). In model 1, an increase of 1% in the number of judges increases the output by 0.04%, a value consistent with model 4. The number of support staff, controlled for the number of judges, raises it almost twice as much, 0.08%. Together, the coefficients related to human resources reach 0.12.

New and pending cases are related to the judge’s workload. An increase of 1% in new cases increases the output by 0.76%. For pending cases, the percentage is lower, 0.06%, and this even has a negative sign in some of the models. Together, the coefficients related to workload amount to 0.84. These results are consistent with of other studies (Beenstock and Haitovsky, 2004; Dimitrova-Grajzl et al., 2012; Lewin, Morey and Cook, 1982; Rosales-López, 2008; Schwengber, 2006), indicating that an increase in the judge’s workload can lead to improved performance, a result that confirms hypothesis H₁. Certainly, the impact of an increase in the judge’s workload will have a limit beyond which there may be decreasing performance. Examination of this phenomenon would typically require longitudinal studies. It is observed that performance increased over time, as indicated by the time-related variables.

The advantage of the stochastic frontier model over DEA is that the former provides a one-step test of individual factors that affect the variation in inefficiency in the labor judiciary. The variable size is relevant for assessing the inefficiency of courts. Large courts seem to be able to reduce inefficiency (-2.07%), unlike small courts (0.4%), a result also observed by Schwengber (2006), which confirms hypothesis H₂.

The electronic process index was not statistically significant for the proposed models. Dummy variables, representing the courts with an electronic process index equal to zero, i.e., that have not adopted the innovation (N_adopt), up to 50% (Adopt) and higher than zero and below 50%, were also included but were not statistically significant, thus failing to confirm hypothesis H₃. This result indicates that using the stochastic frontier technique for the period analyzed, innovation did not translate into decreased court inefficiency. This result may be related to the attempt to standardize routines related to the electronic judicial process on a national scale initiated in 2012, without the elapsed time required to positively impact performance.

Investment in ICT increased the inefficiency of the courts (0.15%), thus failing to confirm hypothesis H₄, and confounded expectations. One possible explanation for this finding is that recent changes that required heavy investments in ICT have not yet had a direct impact on the inefficiency of labor courts. Four factors are commonly listed to explain the paradox related to investment in ICT: resources and performance measurement errors, the time required for adaptation and learning, the redistribution and dissipation of benefits (the investment may not positively impact the sector, although it benefits specific organizations), and problems in the management of IT resources (Brynjolfsson, 1993).

Another possible factor to explain this result is the difference between investment in more commonly used technologies and technologies specifically developed for the organization’s needs. The latter investment is related to an increase in performance. Similarly, the information processing systems specific to the formal and informal context of the organization have the potential to create a sustainable competitive advantage (Barney, 1991).

The workload is statistically significant and has a positive impact in model 1 (0.1%). This relationship is not observed in the other models and is negative for model 4. According to Beenstock and Haitovsky (2004), it is to be expected that the pressure on judges caused by an increased workload will decrease inefficiency. On the one hand, as discussed above, new and pending cases exert pressure for increased production. On the other hand, the variable Workload, in addition to new and pending cases, increases the amount of new and current resources. This increase in resources may impair the performance of the courts. Investment in training (IE), as expected, helps to reduce inefficiency in models 2, 3, and 4, where it was possible to include this variable. The results confirm hypothesis H₅.

The parameter γ has the function of testing the model. It was only statistically significant and close to 1 (0.982) in model 1, which means that 1) the average regression models were not suitable for analyzing the empirical data; 2) the majority of the residual error variation refers to inefficiency; and 3) the random error is close to zero.

5. Conclusions

The objective of the study was to develop and validate theoretical and empirical models that explain the relationships between resources, innovation, and performance in the judiciary and to measure the performance of the labor courts through Malmquist indices and stochastic frontier analysis. The data panel that was used covered 24 Brazilian labor courts between 2003 and 2013, comprising the entire period available in the Justice in numbers database.

Two methods were used, taking advantage of both parametric and non parametric approaches. The Malmquist indices indicate improvement in the index related to innovation adoption in 16 courts, whereas the other seven courts failed to improve performance as a result of innovation adoption. For the period 2003 to 2013, labor courts had more variation in performance related to innovation adoption than related to technical variation. Comparing the calculated indices with the resource variables, a significant and positive relationship was found between the index related to innovation adoption and the variables investment in training and electronic process index, while there was a negative correlation with investment in ICT.

The stochastic frontier analysis confirmed three of the five hypotheses, indicating the following: workload positively impacted performance, with coefficients approximately seven times greater than those related to human resources; court size impacts efficiency; and investment in training helped reduce inefficiency in all models where its inclusion was possible. The following hypotheses were not confirmed: the electronic process index was not statistically significant, and investment in ICT contributed to increased inefficiency, indicating that innovation adoption does not necessarily translate into performance gains and that investments in ICT made because of the standardization phase initiated in 2012 have not yet impacted performance.

The present study fills a gap in the research, develops and tests theoretical and empirical quantitative models of a panel of courts to explain the relationships between resources, innovation and performance in the judiciary. Therefore, the results contribute to knowledge that can planning and resource allocation in the courts, and at a broader level can support policy making for courts. Given that the development and adoption of innovation are an on-going process, these results represent important feedback to those responsible for judicial management and innovation in courts.

The study was limited to data available in a database available in Brazil. The Vtec index analyses performance as a function of innovation adoption but does not specifically identify which innovation and to what extent this contribution occurs. Consequently, other innovations not discussed in this study may influence performance, given that the electronic process index was statistically positive, although low (0.225), indicating that other variables, i.e. other aspects of court professionalization, not studied may impact the Vtec index. Investment in training was similarly significantly positively correlated, although low (0.226). this means that other variables may help to explain performance. The variable investment in ICT covers large investments over the entire cycle, including the acquisition, development, and maintenance of technologies. The variables are available only at the aggregate level of the court.

The use of variables at the chamber level is suggested for future studies. On a micro level, the models may show relationships that are undetected at the aggregate court level, as examined in this study. Analysis of the optimum judge workload, beyond which there is declining performance, is also suggested. Moreover, given the complex nature of measuring the performance of courts, other approaches, for example, the development of a scale to evaluate innovation and performance through latent, not directly observable variables, such as orientation toward innovation, and a more flexible and less formal management model, may shed new light to the court management theme.

References

• ABRAMO, Claudio W. Tempos de espera no Supremo Tribunal Federal. Revista Direito GV, v. 6, n. 2, p. 423-441, 2010.
• ANDERSON-IV, Robert. Distinguishing judges: an empirical ranking of judicial quality in the United States Court of Appeals. Missouri Law Review, v. 76, n. 2, p. 315-348, 2011.
• ARAGÃO, Cecília V. Fatores agilizadores e restritivos à atuação da justiça do trabalho: um estudo exploratório. Rev. Adm. Pública, v. 31, n. 4, p. 183-215, 1997.
• BARNEY, Jay B. Firm resources and sustained competitive advantage. Journal of Management, v. 17, n. 1, p. 99-120, 1991.
• BATTESE, George E.; COELLI, Tim J. A model for technical inefficiency effects in a stochastic frontier production function for panel data. Empirical Economics, v. 20, n. 2, p. 325-332, 1995.
• BEENSTOCK, Michael; HAITOVSKY, Yoel. Does the appointment of judges increase the output of the judiciary? International Review of Law & Economics, v. 24, n. 3, p. 351-369, Sept. 2004.
• BOYNE, George A. Explaining public service perfomance: does management matter?. Public Policy and Administration, v. 19, n. 4, p. 100-117, 2004.
• BOYNE, George A. et al.Explaining the adoption of innovation: an empirical analysis of public management reform. Environment and Planning C: Government and Policy, v. 23, n. 3, p. 419-435, 2005.
• BRYNJOLFSSON, Erik. The productivity paradox of information technology: review and assessment. Communication of the ACM, v. 36, n. 12, p. 67-77, 1993.
• BUSCAGLIA, Edgardo; ULEN, Thomas. A quantitative assessment of the efficiency of the judicial sector in Latin America. International Review of Law & Economics, v. 17, n. 2, p. 275-291, 1997.
• CASTRO, Alexandre S. Indicadores básicos e desempenho da Justiça Estadual de primeiro grau no Brasil. Brasília: Ipea, 2011.
• CHARNES, Abraham; COOPER, William W.; RHODES, Edwardo. Measuring the efficiency of decision making units. European Journal of Operational Research, v. 2, n. 6, p. 429-444, 1978.
• CHOI, Stephen J.; GULATI, Mitu; POSNER, Eric A. How well do measures of judicial ability predict judicial performance? A case study using securities class actions. International Review of Law & Economics, v. 33, p. 37-53, 2013.
• CHRISTENSEN, Robert K; SZMER, John. Examining the efficiency of the U. S. courts of appeals: pathologies and prescriptions. International Review of Law & Economics, v. 32, n. 1, p. 30-37, 2012.
• CNJ. Justiça em números 2013: ano-base 2012. Brasília: {s.n.}, 2013. Available at: <Available at: http://www.cnj.jus.br/images/pesquisas-judiciarias/Publicacoes/relatorio_jn2013.pdf >. Acessed on: 25 Feb. 2015.
  » http://www.cnj.jus.br/images/pesquisas-judiciarias/Publicacoes/relatorio_jn2013.pdf
• CNJ. Justiça em números 2014: ano-base 2013. Brasília: s.n., 2014. Available at: <Available at: ftp://ftp.cnj.jus.br/Justica_em_Numeros/relatorio_jn2014.pdf >. Acessed on: 28 Feb. 2015.
  » ftp://ftp.cnj.jus.br/Justica_em_Numeros/relatorio_jn2014.pdf
• COOPER, William W.; SEIFORD, Lawrence M.; TONE, Kaoru. Data envelopment analysis: a comprehensive text with models, applications, references and DEA-Solver software. 2. ed. New York: Springer Science, 2007.
• COSTA, Paulo R. S. Justiça do trabalho: descentralização e informatização. Revista Eletrônica Parlatorium, v. 1, n. 1, p. 1-7, 2008.
• COSTA, Silvia G. et al. Mensuração da carga de trabalho de magistrados: uma análise comparativa do estudo realizado no Tribunal de Justiça do Estado do Rio Grande do Sul com experiências internacionais. Revista da Faculdade de Direito da Universidade Federal do Rio Grande do Sul, v. 26, n. 221-255, p. 1-30, 2006.
• CRUNKILTON, Dhira D. Staff and client perspectives on the journey mapping online evaluation tool in a drug court program. Evaluation and Program Planning, v. 32, n. 2, p. 119-128, 2009.
• DAMANPOUR, Fariborz; SCHNEIDER, Marguerite. Characteristics of innovation and innovation adoption in public organizations: assessing the role of managers. Journal of Public Administration Research and Theory, v. 19, n. 3, p. 495-522, 2009.
• DAMANPOUR, Fariborz; WISCHNEVSKY, Daniel. Research on innovation in organizations: Distinguishing innovation-generating from innovation-adopting organizations. Journal of Engineering and Technology Management, v. 23, n. 4, p. 269-291, 2006.
• DEYNELI, Faith. Analysis of relationship between efficiency of justice services and salaries of judges with two-stage DEA method. European Journal of Law and Economics, v. 34, n. 3, p. 477-493, 2012.
• DIAS JÚNIOR, Nélio S. A garantia da razoável duração do processo. In: WAMBIER, Teresa A. A. et al. Reforma do judiciário: primeiras reflexões sobre a emenda constitucional n. 45/2004. São Paulo: Revista dos Tribunais, 2004. p. 59-62.
• DIMITROVA-GRAJZL, Valentina et al. Court output, judicial staffing, and the demand for court services: evidence from Slovenian courts of first instance. International Review of Law & Economics, v. 32, n. 1, p. 19-29, 2012.
• FÄRE, Rolf et al. Productivity growth, technical progress, and efficiency change in industrialized countries. The American Economic Review, v. 84, n. 1, p. 66-83, 1994.
• FOLEY, Paul; ALFONSO, Ximena. E-Government and the transformation agenda. Public Administration, v. 87, n. 2, p. 371-396, 2009.
• HARA, Noriko. Information technology support for communities of practice: how public defenders learn about winning and losing in court. Journal of The American Society for Information Science and Technology, v. 58, n. 1, p. 76-87, 2007.
• JOIA, Luiz A. Governo eletrônico e capital intelectual nas organizações públicas. Rev. Adm. Pública, v. 43, n. 6, p. 1379-1405, 2009.
• JOIA, Luiz A. The impact of government-to-government endeavors on the intellectual capital of public organizations. Government Information Quarterly, v. 25, n. 2, p. 256-277, 2008.
• KITTELSEN, Sverre A. C.; FØRSUND, Finn R. Efficiency analysis of Norwegian district courts. Journal of Productivity Analysis, v. 3, n. 3, p. 277-306, 1992.
• KODDE, David A.; PALM, Franz C. Wald criteria for jointly testing equality and inequality restrictions. Econometrica, v. 54, n. 5, p. 1243-1248, 1986.
• LEE, Geunjoo; PERRY, James L. Are computers boosting productivity? A test of the paradox in state governments. Journal of Public Administration Research and Theory, v. 12, n. 1, p. 77-102, 2002.
• LEWIN, Arie Y.; MOREY, Richard C.; COOK, Thomas J. Evaluating the administrative efficiency of courts. Omega The International Journal of Management Science, v. 10, n. 4, p. 401-411, 1982.
• MALMQUIST, Sten. Index numbers and indifference surfaces. Trabajos de Estadística, v. 4, n. 2, p. 209-242, 1953.
• MCKECHNIE, Dougal. The use of the internet by courts and the judiciary: findings from a study trip and supplementary research. International Journal of Law and Information Technology, v. 11, n. 2, p. 109-148, 2003.
• MITSOPOULOS, Michael; PELAGIDIS, Theodore. Does staffing affect the time to dispose cases in Greek courts? International Review of Law & Economics, v. 27, n. 2, p. 219-244, 2007.
• OSBORNE, David; GAEBLER, Ted. Reinventing government: how the entrepreneurial spirit is transforming the public sector. New York: Plume, 1993.
• PEDRAJA-CHAPARRO, Francisco; SALINAS-JIMENEZ, Javier. An assessment of the efficiency of Spanish Courts using DEA. Applied Economics, v. 28, n. 11, p. 1391-1403, 1996.
• RAMSEYER, Mark J. Talent matters: judicial productivity and speed in Japan. International Review of Law & Economics, v. 32, n. 1, p. 38-48, 2012.
• RAUEN, Andre T. Desafios da avaliação em políticas de inovação no Brasil. Revista do Serviço Público, v. 64, n. 4, p. 427-445, 2013.
• ROGERS, Everett M. Diffusion of innovations 5. ed. New York: The Free Press, 2003.
• ROSA, João; TEIXEIRA, Claudio; PINTO, Joaquim S. Risk factors in e-justice information systems. Government Information Quarterly, v. 30, n. 3, p. 241-256, 2013.
• ROSALES-LÓPEZ, Virginia. Economics of court performance: an empirical analysis. European Journal of Law and Economics, v. 25, n. 3, p. 231-251, 2008.
• SAMAN, Wan S. Wan Mohd; HAIDER, Abrar. E-Shariah in Malaysia: technology adoption within justice system. Transforming Government: People, Process and Policy, v. 7, n. 2, p. 256-276, 2013.
• SCHNEIDER, Martin R. Judicial career incentives and court performance: an empirical study of the German Labour Courts of Appeal. European Journal of Law and Economics, v. 20, n. 2, p. 127-144, 2005.
• SCHWENGBER, Silvane. Mensurando a eficiência no sistema Judiciario: métodos paramétricos e não paramétricos. 2006. Thesis (PhD) - Faculdade de Economia, Administração, Contabilidade e Ciência da Informação e Documentação, Universidade de Brasília, Brasília.
• SMYTH, Russell; BHATTACHARIA, Mita. How fast do old judges slow down? A life cycle study of aging and productivity in the Federal Court of Australia. International Review of Law & Economics, v. 23, n. 2, p. 141-164, 2003.
• SOARES, Yuri; SVIATSCHI, Maria M. The impact of modernization on court efficiency in Costa Rica. OVE working papers 0610, Inter-American Development Bank, Washington, 2010.
• SOLOW, Robert M. We’d better watch out. New York Times, 1987. p. 36.
• SOUSA, Marcos M.; GUIMARAES, Tomas A. Inovação e desempenho na administração judicial: desvendando lacunas conceituais e metodológicas. Revista de Administração e Inovação, v. 11, n. 2, p. 321-344, 2014.
• SOUSA, Marcos M.; GUIMARAES, Tomas A. The adoption of innovations in Brazilian labor courts from the perspective of judges and court managers. Revista de Administração, v. 52, n. 1, p. 103-113, 2017.
• STRATOPOULOS, Theophanis; DEHNING, Bruce. Does successful investiment in information technology solve the productivity paradox? Information & Management, v. 38, n. 2, p. 103-117, 2000.
• TST . Varas instaladas e não instaladas 2015. Available at: <Available at: http://www.tst.jus.br/instaladas-e-nao-instaladas >. Accessed on: 24 Sept. 2015.
  » http://www.tst.jus.br/instaladas-e-nao-instaladas
• TULKENS, Henry. On FDH efficiency analysis: some methodological issues and applications to retail banking, courts, and urban transit. The Journal of Productivity Analysis, v. 4, n. 1, p. 183-210, 1993.
• VELICOGNA, Marco. Justice systems and ICT: what can be learned from Europe? Utrecht Law Review, v. 3, n. 1, p. 129-147, 2007.
• VELICOGNA, Marco; ERRERA, Antoine; DERLANGE, Stéphane. Building e-justice in Continental Europe: the TéléRecours experience in France. Utrecht Law Review, v. 9, n. 1, p. 38-59, 2013.
• VELICOGNA, Marco; ERRERA, Antoine; DERLANGE, Stéphan e. e-Justice in France: the e-Barreau experience. Utrecht Law Review, v. 7, n. 1, p. 163-187, 2011.
• YEUNG, Luciana L.; AZEVEDO, Paulo F. Measuring efficiency of Brazilian courts with data envelopment analysis (DEA). Journal of Management Mathematics, v. 22, p. 343-356, 2011.
• ZHU, Joe. Quantitative models for performance evaluation and benchmarking: data envelopment analysis with spreadsheets. 2. ed.New York: Springer Science , 2009.

• 3
  Research funded by Capes (first author’s grant) and CNPq (research project by the second author).
• 4
  The authors are grateful for the criticisms and suggestions made by anonymous reviewers of RAP, who contributed to the improvement of the quality of the text.

Publication Dates

• Publication in this collection
  May-Jun 2018

History

• Received
  08 Mar 2017
• Accepted
  03 Mar 2018