Innovation output indicators: relevance for policies, the EU 2020 indicator and an alternative proposal

ABSTRACT

This article introduces a new family of innovation output indicators as an alternative to the EU 2020 indicator. The latter is a composite indicator that bears a complex and weak relation to the actual innovation output of economies. The proposed family of indicators (DINNOV) estimates the participation in the economy of enterprises that are both dynamic (or high-growth) and innovative. Its indicators take advantage of both the tradition of indicators produced from enterprise data collected by surveys of innovation and of indicators of business demography and entrepreneurship. Values of DINNOV indicators for Brazil are computed as a proof of concept. A proxy of the DINNOV indicators – the DINNOV-Simplex – is estimated for Brazil and 17 European economies. The new indicators avoid several drawbacks of the EU 2020 innovation output indicator and are easier to communicate to policymakers and the general public.

KEYWORDS:
Innovation output; Innovation indicator; DINNOV; Surveys of innovation; Innovation policy; High-growth enterprises; Brazil; European Union

1. Introduction

The understanding of how science, technology and innovation interact with each other and influence the development of economies and societies has evolved significantly over the last decades. Innovation has come to play a central role in this new understanding. New statistics and indicators of innovation started to be regularly collected in a large number of countries. However, many of the implications of that evolution on policymaking have not yet come about. One of the manifestations of this disconnect is the fact that R&D indicators associated with the old linear model (STOKES, 1997) still have a preeminent role in policymaking and policy evaluation.

That perception is developed and substantiated in the second section of this article, entitled “We see innovation indicators everywhere, but in policy”. The section recalls the European Commission’s decisions to place innovation at the heart of its 2020 strategy, to develop a new innovation indicator focusing on outputs and impacts and to attach to this indicator a key role in its strategy for benchmarking the performance of leading industrial countries (EUROPEAN COMMISSION, 2010a, p. 2, 29-30). The European Commission’s intention was to be able to set targets measured in terms of innovation output in a similar way to the targets they had already set regarding R&D expenditures as a percentage of the GDP.

The second section also stresses the perception that economies that are far from the technological frontiers, such as Brazil, could have a special interest in an innovation output indicator similar to the one proposed by the European Commission. In the case of Brazil, for instance, there is a strong disconnect between its fast-growing scientific production and its dismal innovation output. An important reason for such a problem is the prevalence in Brazil of policies and policy instruments inspired by the simplistic conviction in an almost direct relationship between the amount of resources invested in R&D and their results in terms of technological innovation (VIOTTI, 2013). In such an environment, an innovation output indicator would be useful in the effort to develop a more effective innovation policy.

The third section presents the EU 2020 innovation indicator introduced by the European Commission and the methodology used for its computation. It also develops an assessment of the extent to which the proposal fulfilled the expectations that drove its creation. The overall conclusion is that the EU 2020 innovation indicator lacks the necessary requirements to play the role it was developed for.

As a response to that assessment, in the fourth section, we introduce a new family of innovation output indicators named DINNOV. These indicators measure the participation in the economy of enterprises that are simultaneously dynamic and innovative, where dynamic stands for enterprises with high growth measured in terms of employment. The concept of the DINNOV indicators was tested using microdata from Brazil for the period 2008-2012. A proxy of the DINNOV-Enterprise indicator, called DINNOV-Simplex, was computed for 17 European countries and Brazil. The test was useful to confirm the possibility of computing the DINNOV indicators using the available data from surveys of innovation and business registries and to show that they are much easier to communicate and to be understood by policymakers and the public in general than the EU 2020 indicator.

The last section presents the main conclusions of the article.

2. We see innovation indicators everywhere, but in policy

Most scientists, policymakers and analysts of Science, Technology and Innovation (ST&I) policy ascribe to R&D indicators a commanding role in their analysis, and this has serious consequences, especially for economies that are far from the technological frontier.

Concrete evidence of this perceived focus is the fact that many countries set policy targets in terms of R&D expenditure and that this type of target usually comes to be a focal point of effort and attention. In this respect, Carvalho (2018, p. 373) stated that “it is currently common for governments to set R&D objectives and many countries have done so over the past 50 years”. Their major motivation appears to be the conviction that a country’s level of R&D explains much of its innovation capability (CARVALHO, 2018, p. 384). The author managed to collect information about R&D intensity target setting and on how effective they were in 45 economies and concluded that “the popularity of the R&D intensity indicator remains high despite the complete lack of effectiveness of R&D policy based on R&D intensity targets. (…) It is a revival of a 1960s policy based on similar arguments but completely different context” (CARVALHO, 2018, p. 373).

This revival seems paradoxical when compared to the huge evolution that happened in the economics of innovation and in its expected implications for ST&I policies.

During the last three decades, a systemic approach, developed under the Schumpeterian tradition, has gained increasing traction among scholars and scientists engaged in the study of the determinants of innovation and of the technological development of countries. In the national innovation systems (NIS) approach, R&D institutions and efforts play a relevant role, but enterprises are the central players. However, enterprises do not innovate in isolation. They do so in the context of a network of direct and indirect relationships with other enterprises, public and private research infrastructure, educational institutions, the normative system and the economic, political and cultural environment in which they are embedded (FREEMAN, 1987; LUNDVALL, 1988, 1992; NELSON, 1988, 1993). The NIS approach diffused quickly among policy-makers in a growing number of countries and in international institutions such as the Organisation for Economic Co-operation and Development (1997, 1999, 2002), the European Commission and the United Nations Conference on Trade and Development (2011).

Alongside the development and diffusion of the NIS approach, a comprehensive framework for the production of statistics on innovation was developed. The initial result of this work was published in 1992 as the first edition of the Oslo Manual: Proposed Guidelines for Collecting and Interpreting Technological Innovation Data (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 1992). It is true that the manual on innovation indicators appeared approximately thirty years later than the Frascati Manual on R&D indicators, but thirty years had already passed since the first edition of the Oslo Manual. Three subsequent revised and improved editions were introduced, and a huge experience of production and use of innovation data and indicators was accumulated during that period. Surveys of innovation guided by the Oslo Manual have been carried out in more than 80 countries (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 2018, p. 3).

Despite such a remarkable evolution, policy-makers and policy analysts still continue to ground their work essentially on R&D data and on the linear model. This seems to have been the case even in the European Union (EU), where the NIS approach emerged and is most influential, innovation data are available and where innovation has a very important place in the policy discourse. Based on a questionnaire answered by 18 EU countries, Edquist (2014, p. 10) concluded that “the use of the innovation systems approach for actual policy purposes is still often a matter of lip service; it is mainly used only by name. The content of innovation policies is still dominated by the linear model.”

Recognizing the limitations of previous ST&I policies and the utmost importance of developing an actual innovation policy, the European Commission decided to place “innovation … at the heart of the Europe 2020 strategy” (EUROPEAN COMMISSION, 2010a, p. 2).

The same document that defines such a broad policy objective explicitly recognizes the need to develop a new indicator to monitor the progress of European policy toward tackling that challenge by stating that “[p]rogress towards the Innovation Union should be measured … by two headline indicators: the R&D investment target and a new Innovation indicator, as requested by the European Council” (EUROPEAN COMMISSION, 2010a, p. 29). In accordance with this guidance, “a High-Level panel of leading business innovators and economists was set up to identify possible indicators […] focusing on outputs and impacts and ensuring international comparability” (EUROPEAN COMMISSION, 2010a, p. 29).

The panel’s recommendation supported the development and use of an indicator that measures innovative fast-growing enterprises (EUROPEAN COMMISSION, 2010a, p. 29). This option was recommended because it was associated with the dynamism of the economy in terms of economic growth and job creation. It was also supported because it is output-oriented rather than input-oriented, as R&D indicators are. This indicator was also seen as capable of reflecting the impact of the systemic framework conditions on innovation, which can be influenced by European and national policies. (EUROPEAN COMMISSION, 2010a, p. 29; 2010b)

The European Union's innovation strategy for 2020 attached a key role to this new indicator in its strategy of benchmarking the performance of its main trading partners (EUROPEAN COMMISSION, 2010a, p. 30). In other words, it was the aim of the European Union policy to achieve performance standards in terms of innovative fast-growing enterprises (as measured by the new indicator) similar to those that prevailed in the economies of their main competitors.

A new innovation output indicator such as the one discussed here has an importance for innovation policy that goes far beyond the borders of the European Union. The possible existence of an innovation output indicator could draw attention to the limited effects of conventional science and technology (S&T) policy and could also help to create an environment more conducive to the implementation of actual innovation policies.

In the case of Brazil, for instance, innovation came to play an increasingly important role in ST&I policies, but their practice continues to coexist with methods of the old tradition of strictly S&T policy (VIOTTI, 2008a; 2008b). The two most recent Brazilian ST&I policy plans – National Strategy of Science, Technology and Innovation (ENCTI, in its Portuguese acronym) 2012-2015 and 2016-2022 (BRASIL, 2012, 2016) – express commitment to foster innovation, but R&D instruments and indicators still prevail. The indicators set for the follow-up processes of those plans give an important clue about the relative importance R&D still has. ENCTI 2012-2015 set 16 indicators for its follow-up. Two of them were related to innovation, six to R&D, and almost all of the others were related to human resources. ENCTI 2016-2022 set 10 indicators for its follow-up. The same innovation and R&D-related indicators of the first ENCTI remained, and the number of human resource-related indicators was reduced to only two.

Currently, a consensus prevails among a large part of Brazilian scientists, policymakers, economists and businesspeople: scientific production is advancing at a fast pace in Brazil, but innovation is not. A feeling of this disconnect is usually captured from the comparison between indicators for scientific production and those of utility patents, even though one must take into consideration the limitations to the usual inferences made from patent data to evaluate the strength of innovation processes. Brazil’s share of world’s scientific production measured by the number of papers published in peer-reviewed journals by residents in Brazil grew from 0.77% in 1996 to 2.60% in 2019.¹ During the same period, the share of utility patent applications filed by nonresidents in the US Patent and Trademark Office (USPTO) credited to residents in Brazil evolved from 0.13% in 1996 to 0.23% in 2019.² In 2019, Brazil’s share of world scientific production was more than 11 times larger than that country’s share of utility patents filed by nonresidents in the USPTO.

This kind of disconnect could be related to a structural feature of countries such as Brazil. The relatively fast advancement of scientific knowledge production could be less meaningful for innovation and technological development in countries such as Brazil than it usually is for more advanced economies. The national innovation systems of more advanced economies and their enterprises are expected to be more able to profit from advances in scientific knowledge than the systems and enterprises of countries such as Brazil, which could be better characterized as National Systems of Technological Learning than as truly National Innovation Systems (VIOTTI, 2002). Therefore, policies wittingly inspired by the linear model or that follow its main traits by inertia would likely be less helpful for less advanced economies. Hence, the use of an innovation output indicator to set policy targets or to assess policy outcomes should be very much welcomed in countries such as Brazil.

The next section will present the new European Union 2020 innovation output indicator methodology and develop an assessment of how far the proposal came to fulfill the recommendations of the high-level panel and the expectations that drove its initial commissioning by the European Council.

3. The EU 2020 innovation output indicator

The new innovation output indicator introduced by the European Commission in 2013 (2013a, 2013b, 2013c) is a composite indicator that aggregates the following components (VÉRTESY, 2017):

1. 1

   PCT – Technological innovation as measured by patent applications filed under the Patent Cooperation Treaty per billion GDP (in PPS €)

2. 2

   KIABI – Employment in knowledge-intensive activities as a percentage of total employment

3. 3

   COMP – Competitiveness of knowledge-intensive goods and services, based on their contributions to exports

   • 3.1 GOOD – Exports of medium and high technology products as a share of total product exports, and

   • 3.2 SERV – Knowledge-intensive services exports as a percentage of total services exports.

4. 4

   DYN – Employment in fast-growing enterprises of innovative sectors as a percentage of total employment

The composite indicator is obtained after a complex process of computation (VÉRTESY; DEISS, 2016, p. 17, and VÉRTESY, 2017, p. 26):

• The value of each component for every country is normalized by subtracting the mean of the values of the indicator for all countries and dividing the result by the standard deviation of its distribution, both for the reference year (z score).

• “The obtained z scores are rescaled to a positive range using the following formula: z*1.5+5.”

• The z score normalized components are aggregated according to the formula$I=w_{1}PCT+w_{2}KIABI+w_{3}COMP+w_{4}DYN$, where w₁, w₂, w₃, and w₄ are the weights of the component indicators.

• The weights are obtained with the objective of making the composite indicator statistically equally balanced in its components.

• The scores of the composite indicator obtained with these calculations are then renormalized to make them reference the value the composite indicator obtained in 2011 for the European Union aggregate equal to 100.

• The aggregation described above is carried out for two different datasets. The first one is used essentially for comparisons between EU countries.

• A second aggregation is done with the purpose of being used for comparisons between the EU aggregate and selected international benchmark countries.

• The “composite scores obtained from the two datasets are not directly comparable with one another”.

The merit of the chosen components and composite indicators is the fact that they use data that are relatively easy to obtain, but they have important drawbacks.

PCT is an indicator of invention rather than innovation. It could be used as an innovation indicator, but it is a highly imperfect indicator because patents propensity varies significantly among economic activities and technical fields, a very large proportion of patents never come to be industrially applied and innovations are not necessarily associated with patented inventions (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 2009, p. 26-28).

KIABI would be better characterized as an input indicator. Knowledge intensive activities (KIA) are those activities in which at least 33% of employees had a tertiary degree, but they are not computed for every country and year. They are preselected by a static and standardized classification. Consequently, the indicator does not take into consideration variations in knowledge-intensive activities between countries and over time.

GOOD gauges the strength of the innovation process of a country by the performance of its exports of high-tech and medium-tech products. Nonetheless, the classification of technology intensity was originally built taking as a reference for the intensity of R&D expenditures performed in each activity. It therefore shows a facet of input rather than of output indicator, and it is related to the old R&D indicators from which the new innovation output indicator intends to move away. Moreover, it reflects just the conditions of a specific set of countries in a given time in the past. Furthermore, the rapid and strong development of global chains of value of the past decades undermined the ability of the GOOD indicator to measure what it is supposed to gauge.

High levels of SERV could be viewed as an indirect consequence of the country’s innovation process. However, knowledge-intensive services are not computed for every country and year; they are defined by a standardized and static classification supported by the same criteria used by the KIABI classification and are related to inputs of skilled labor.

The original purpose of the DYN indicator – to measure business dynamism in fast-growing and innovative enterprises – would constitute a true innovation output indicator. However, difficulties in computing DYN at an enterprise level led the proponents of the indicator to once again use a sector-based approach (VÉRTESY, 2017, p. 19-20), similar to what had happened with all the previous component indicators, except PCT. That classification, based on an experimental taxonomy developed by the OECD, ascribed indexes of innovativeness to manufacturing and services sectors based on their respective coefficients of innovativeness (“cis”) and knowledge intensity (“kia”) (VÉRTESY, 2017, p. 20). The cis score was obtained using pooled European Community Innovation Survey (CIS) microdata, and the kia scores were obtained according to the shares of tertiary educated employees in every sector (VÉRTESY, 2017, p. 20). The proportion of total employment that comes from fast-growing enterprises in the manufacturing and services activities thus classified among the most innovative sectors was defined as the DYN indicator.

As it abandoned the enterprise level approach, the DYN indicator became insensitive to differences in the innovation process through time and across countries in the same sectors. Additionally, the same way to infer the knowledge intensity of sectors that is the basis for the KIABI indicator and the kia score is once again inserted in the DYN. In this respect, an input related to the size of the skilled labor force was also inserted into the DYN.

Each of the component indicators analyzed above contributes to a greater understanding of the innovation processes of countries in some respect, but most of them are associated with serious drawbacks that undermine their ability to infer the intended innovation output. Surprisingly, the results of countries’ surveys of innovation were utterly disregarded and replaced by static and uniform sectoral classifications in all component indicators, except PCT. As a consequence, the proposed innovation output indicator is essentially insensitive to differences between countries’ innovation processes other than those that can be captured by changes in their sectoral structures (VIOTTI et al., 2015, p. 103-108; JANGER et al., 2017). Moreover, the advance of globalization, digitalization and the emergence of new business models exacerbate this problem because they are progressively eroding firms’ similarity within each economic activity. This constitutes an additional reason for the preference that should be given to indicators based on firms’ microdata, such as the one proposed in the next section.³

Furthermore, the complex way in which the component indicators are integrated to obtain the composite indicator certainly distances the EU 2020 innovation output indicator from an easy and intuitive understanding by policy-makers and analysts.

The composite indicator is estimated after the component indicators go through a series of steps in a long statistical assembly line of transformations. This complex process makes it difficult to compare the values of the composite indicator in different years⁴, to trace the impact of its components, to understand its meaning and to communicate its results to an audience that goes beyond the community of statistics producers.⁵

As analyzed in the previous paragraphs, the selected component indicators and the method employed for their aggregation jeopardized the very purpose that led to the creation of the innovation output indicator. It is no wonder that the proposed indicator seems to have failed to meet the high expectations placed on it by the European Commission itself and by those that see an innovation output indicator as an important tool to help steer ST&I policies toward genuine innovation policies. The frustration of those expectations encouraged the authors to explore the possibility of developing innovation output indicators inspired by the most promising component indicator, the DYN, avoiding the need to resort to sectoral classifications. The main results of those efforts are synthetized in the next section.

4. The DINNOV family of indicators

The way forward in the quest for a genuine innovation output indicator seems not to rely either on composite indicators or on the assumption of the existence of a similar and stable innovative behavior for enterprises of each activity in all countries. The family of innovation output indicators proposed here – DINNOV – aspire to advance a step forward in that quest. It is composed of a set of similar ways to measure the participation in the economy of enterprises that are simultaneously dynamic and innovative, where dynamic stands for high-growth enterprises.

Figure 1, a representative diagram of enterprises according to their attributes regarding high growth and innovation, presents an easy way to grasp the concept of DINNOV. Among all enterprises, represented by the larger circle, there are two smaller circles representing innovative and high-growth enterprises. The intersection of these smaller circles depicts enterprises that are simultaneously innovative and high-growth. The DINNOV indicators are measurements of the proportional contribution of the set of enterprises in that intersection to all enterprises in terms of number of enterprises, number of employees and value added.

FIGURE 1
Representative diagram of enterprises according to their attributes regarding high growth and innovation.

The DINNOV family of indicators seems closer to meeting the European Commission’s expectation of providing “a good measure of the dynamism of the economy; captures an important part of our economy where growth and jobs will need to come from; it is output-orientated and reflects the impact of the framework conditions on innovation, which policy-makers … can influence” (EUROPEAN COMMISSION, 2010a, p. 29). Furthermore, the indicators’ association of innovation with growth is supported by the realization that innovation is a very important engine for growth (by contributing to the acceleration of enterprises’ growth and to the creation of new and dynamic enterprises and activities) and that growth is also a stimulating factor for innovation. In other words, there is a virtuous circle of innovation and growth.⁶

The DINNOV family of indicators takes advantage of two different strains of the literature on indicators. The first is the already mentioned tradition of producing innovation indicators from data on enterprises collected by surveys of innovation grounded on the methodology established by the Oslo Manual. The other is the more recent development of indicators of entrepreneurship that follows the guidelines set by the “Eurostat-OECD Manual on Business Demography Statistics - 2007” (EUROPEAN COMMISSION, 2008). Business registers are used to generate indicators of entrepreneurship.

Dynamic or high-growth enterprises are those (of 10 or more employees) “with average annualized growth greater than 20% per annum, over a three-year period”, where growth is measured by the number of employees (EUROPEAN COMMISSION, 2008, p. 82). Innovative enterprises are considered those “that ha[ve] implemented a new or significantly improved product or process during the period under review” (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 2005, p. 47).⁷

The DINNOV family of indicators comprises 4 measures for the participation in the economy of enterprises that are simultaneously dynamic and innovative.

• DINNOV-Enterprise – Number of dynamic and innovative enterprises as a proportion of the total number of enterprises;

• DINNOV-Employment – Number of employees in dynamic and innovative enterprises as a proportion of the total number of employees;

• DINNOV-Value-Added – The contribution of dynamic and innovative enterprises to the value added as a proportion of the value added by all enterprises; and

• DINNOV-Simplex – Number of dynamic and innovative enterprises as a proportion of the total number of enterprises estimated by the innovation rate times the high-growth rate.

An empirical exercise using microdata from Brazilian surveys of innovation and administrative registries was conducted specifically with the aim of demonstrating the feasibility of DINNOV estimation.⁸

The hardest task is to estimate the hard-core formed by the enterprises that are simultaneously dynamic and innovative, as it requires working with microdata, which is often obtained from survey samples. The identification or inference about that hard-core of enterprises was done in four steps in the Brazilian case.

The first step was the identification of high-growth enterprises. For this purpose, the central registry of business enterprises (CEMPRE) maintained by the Brazilian Institute of Statistics (IBGE) was used.

The second step requires the identification of the enterprises that introduced product or process innovations during the period of reference among the sample of enterprises that answered the questionnaire of the Brazilian survey of innovation (PINTEC).

The third step is to cross-reference information about high-growth enterprises with those on innovative enterprises included in the PINTEC sample. Since the first step identified all high-growth enterprises, it is possible to identify the high-growth ones included in the PINTEC sample that were innovative. This set of enterprises will constitute the basis for the estimations about the innovative and high-growth enterprises as a whole.

The fourth and last step is the expansion of the cross-referenced result achieved in the previous step to the whole subdomain of high growth enterprises according to the coefficients of expansion used in the Brazilian survey of innovation for each economic activity and enterprise class-size.

With that statistical expansion, it is then possible to estimate the number of enterprises that are simultaneously dynamic and innovative, as well as their contribution to employment and to the value added for each of the economic activities included in the scope of the Brazilian innovation surveys. It is thus also possible to estimate the DINNOV-Enterprise, DINNOV-Employment and DINNOV-Value-Added indicators.

The method used to compute the values of the DINNOV indicators described above is, however, constrained by the survey’s periodicity. Innovation surveys are usually collected only every 3 years in Brazil. Additionally, survey results are published no sooner than one year after data collection. To circumvent this limitation, an extrapolation procedure is used to produce preliminary estimates for the DINNOV indicators for years for which the results of innovation surveys are not available. Such an extrapolation is based on the assumption that the most recent available rate of innovation among high-growth enterprises remains unchanged. As the Brazilian central registry allows for the computation of high-growth enterprises every year, preliminary values of DINNOV indicators are estimated by using that extrapolated rate of innovation. When new rates of innovation become available, revised estimates for the DINNOV indicators are computed for the years between surveys based on an interpolation procedure.

Following the methodology described above, the main results achieved by the process of estimating the values of indicators DINNOV-Enterprise, -Employment and -Value-Added for Brazil during the years 2008 to 2012 are presented in Tables 1, 2 and 3.⁹ Table 1 presents the estimates of the number of all enterprises and of the subsets of innovative, high-growth, and simultaneously innovative and high-growth enterprises. The last column of Table 1 presents the proportion of innovative and high-growth enterprises in the total number of enterprises, which is the same as the DINNOV-Enterprise indicator. Table 2 presents the estimates of the number of employees in high-growth, in high-growth and innovative, and in all enterprises. The last column of Table 2 presents the proportion the number of employees in innovative and high-growth enterprises represents in the number of employees in all enterprises, which is the same as the DINNOV-Employment indicator. Table 3 presents the estimates of the value added in high-growth, in high-growth and innovative, and in all enterprises. The last column of Table 3 presents the proportion value-added in innovative and high-growth enterprises represented in the value-added in all enterprises, which is the same as the DINNOV-Value-Added indicator.¹⁰

TABLE 1
Number of high-growth, high-growth and innovative, and of all enterprises; high-growth rate, innovation rate of high-growth enterprises, and DINNOV-Enterprise, Brazil, 2008-2012

Year	Enterprises (Total)	High-Growth enterprises	High-Growth rate	High-Growth and innovative enterprises	Innovation rate of High-Growth enterprises	DINNOV-Enterprise (C/A) or
	(A)	(B)	(B/A)	(C)	(C/B)	(B/A) x (C/B)
2008	106,862	9,279	8.7%	4,058	43.7%	3.8%
2009	100,760	8,574	8.5%	(NA)	46.3%	3.9%
2010	106,740	9,136	8.6%	(NA)	46.3%	4.0%
2011	124,476	9,064	7.3%	4,450	49.1%	3.6%
2012	109,831	9,026	8.2%	(NA)	49.1%	4.0%

• Sources: Special tabulations of CEMPRE/IBGE; PINTEC 2008 and 2011 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2010, 2013a); and Estatísticas de Empreendedorismo 2008, 2010, 2011 and 2012 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2011, 2012, 2013b, 2014a) (Authors' elaboration). Note: Table data refer to activities and size classes of enterprises included in the PINTEC's scope.

TABLE 2
Number of employees in high-growth, in high-growth and innovative, and in total enterprises; and DINNOV-Employment, Brazil, 2008-2012

Year	Employees (Total)	Employees in HG enterprises	Employees in HG enterprises/Total employees	Employees in HG and innov. enterprises	Employees in HG and innov. enterprises/Employees in H-G enterprises	DINNOV-Employment
						(C/A) or
	(A)	(B)	(B/A)	(C)	(C/B)	(C/B) x (B/A)
2008	7,488,783	1,403,225	18.7%	820,503	58.5%	11.0%
2009	7,674,321	1,343,821	17.5%	(NA)	58.1%	10.2%
2010	8,255,385	1,354,872	16.4%	(NA)	58.1%	9.5%
2011	8,604,491	1,385,976	16.1%	800,318	57.7%	9.3%
2012	8,637,168	1,312,638	15.2%	(NA)	57.7%	8.8%

• Sources: Special tabulations of CEMPRE/IBGE; PINTEC 2008 and 2011 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2010 and 2013a); and Estatísticas de Empreendedorismo 2008, 2010, 2011 and 2012 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2011, 2012, 2013b, 2014a) (Authors' elaboration). Note: Table data refer to activities and size classes of enterprises included in the PINTEC's scope.

TABLE 3
Value added in high-growth, in high-growth and innovative, and in all enterprises; and DINNOV-Value-Added, Brazil, 2008-2012

Years	VA in all enterprises (R$ millions)	VA in HG enterprises (R$ millions)	VA in HG enterprises/VA in all enterprises	VA in HG and innov. enterprises (R$ millions)	VA in HG and innov. enterprises/VA in HG enterprises	DINNOV-Value-Added
						(C/A) or
	(A)	(B)	(B/A)	(C)	(C/B)	(C/B) x (B/A)
2008	706,492	99,912	14.1%	74,159	74.2%	10.5%
2009	651,204	76,084	11.7%	(NA)	77.7%	9.1%
2010	804,298	95,148	11.8%	(NA)	77.7%	9.2%
2011	940,597	101,914	10.8%	82,846	81.3%	8.8%
2012	976,120	104,864	10.7%	(NA)	81.3%	8.7%

• Sources: Special tabulations of CEMPRE/IBGE; PINTEC 2008 and 2011 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2010 and 2013a); Estatísticas de Empreendedorismo 2008, 2010, 2011 and 2012 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2011, 2012, 2013b, 2014a); and PIA-Empresa 2011 e 2012 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2013c and 2014b) (Authors' elaboration). Notes: Table data refer to activities and size classes of enterprises included in the PINTEC's scope exclusively selected services. Services were not included because the research project did not have access to the microdata of the Annual Survey of Services, a required condition to compute this indicator for years without innovation surveys. Industrial manufacturing value (IMV) was computed instead of value added (VA) itself because the research project did not have access to the microdata on the Annual Industrial Survey. IMV is obtained by the subtraction of operational costs from the gross value of production. An exercise done for the years with innovation surveys, for which it was possible to compute the VA, showed small differences between the values estimated for VA and for IMV.

The fourth member of the family, DINNOV-Simplex, is a way to compute a proxy of the first indicator, the DINNOV-Enterprise. It is specifically designed for economies for which the microdata necessary for the generation of estimates about dynamic and innovative enterprises are not available. As a way to circumvent this limitation, that proxy uses data that are computed and published regularly for many countries: the innovation rates produced by innovation surveys and the high-growth rates generated by business and entrepreneurship statistics. DINNOV-Simplex is computed by the simple multiplication of those two rates. In the absence of more accurate information on the rate of innovation of high-growth firms, it is assumed that its best possible estimation would be the rate of innovation that prevails for all firms in the economy.

Estimates of the DINNOV-Simplex for Brazil and 17 selected European economies for the period 2008-2014 are shown in Table 4. The values of these rates are shown in Tables A1 and A2 in the Annex to this article.

TABLE 4
DINNOV-Simplex, selected countries, 2008-2014 (%)

Country	2008	2009	2010	2011	2012	2013	2014	2008-2014 (avg.)
Brazil	3.1	2.7	2.7	2.5	2.5	2.2	2.0	2.5
Czechia	2.0	1.3	1.1	1.1	1.7	1.7	1.8	1.5
Denmark	0.8	0.8	0.7	0.5	0.5	0.5	0.5	0.6
Estonia	2.5	1.6	1.2	1.3	1.5	1.0	0.7	1.4
France	2.2	2.1	1.4	1.3	1.4	1.4	1.5	1.6
Hungary	0.9	0.6	0.5	0.6	0.7	0.7	0.7	0.7
Italy	0.9	0.8	0.6	0.6	0.6	0.6	0.5	0.6
Latvia	1.8	1.5	1.2	1.3	2.2	1.8	1.6	1.6
Lithuania	0.5	0.5	0.4	0.5	0.7	1.0	1.4	0.7
Luxembourg	1.8	0.8	1.0	1.1	1.1	1.0	1.0	1.1
Netherlands	1.7	1.9	2.2	2.0	1.9	1.9	1.9	1.9
Portugal	1.6	1.0	0.8	0.7	0.7	0.8	0.8	0.9
Romania	0.2	0.1	0.1	0.1	0.1	0.1	0.0	0.1
Slovakia	2.6	2.8	3.1	2.5	2.0	2.6	2.8	2.6
Slovenia	1.6	0.8	0.6	0.6	0.6	0.6	0.6	0.8
Spain	0.5	0.5	0.4	0.5	0.5	0.6	0.5	0.5
Sweden	1.5	1.6	1.6	1.5	1.5	1.4	1.4	1.5
United Kingdom	1.2	1.2	1.2	1.2	1.2	1.3	1.4	1.2

• Sources: Eurostat, Community Innovation Surveys - CIS <https://ec.europa.eu/eurostat/web/science-technology-innovation/data/database>, accessed 09/03/2020. OECD. Stat, SDBS Business Demography Indicators <https://stats.oecd.org/Index.aspx?DataSetCode=SDBS_BDI_ISIC4#>, accessed 03/03/2020. IBGE, Pesquisas de Inovação - PINTEC <https://sidra.ibge.gov.br/tabela/5453>, accessed 04/27/2020. IBGE, CEMPRE (Authors' estimation of HGR for Brazil) (Authors’ elaboration).

The Brazilian DINNOV-Simplex rates seem to be unexpectedly high for a country without a strong innovation tradition. The average Brazilian DINNOV-Simplex rate for the period 2008-2014 (2.5%) is more than 2 times higher than the mean of all the other countries included in Table 4 (1.1%). As the average Brazilian innovation rate for the period (36.7%) is almost identical to the average of the other countries (36.3%), Brazil’s outstanding performance in that indicator could be essentially explained by Brazil's relatively high rate of high-growth enterprises at the time. The Brazilian average rate of high-growth enterprises (6.9%) was more than three times higher than the average rate of the other countries (1.9%). Such a difference was certainly correlated with the exceptional growth Brazil experienced during the 7 years that followed the start of the 2008 Recession. The average rate of growth of the Brazilian economy in those years (3.1%) was more than 10 times higher than that of the selected European economies included in Table 4 (0,3%).

It is interesting to note that, even in the case of a complete family of DINNOV indicators just for one country and for a short period of time, the estimates of the DINNOV-Enterprises indicator were relatively closer to those of the indicator devised to be its proxy (Figure 2). Obviously, a much larger number of observations will be needed to statistically test how good a proxy the DINNOV-Simplex is for the DINNOV-Enterprise.

FIGURE 2
DINNOV -Enterprise, -Employment, -Value-Added and –Simplex, Brazil, 2008-2012.

Another interesting aspect brought about by the analysis of Figure 2 is the fact that the contribution of the dynamic and innovative enterprises to the Brazilian economy measured in terms of employment and value added – DINNOV-Employment and DINNOV-Value-Added – is more than two times higher than their contribution to the population of enterprises. Curiously, the share of high-growth and innovative enterprises in the total number of enterprises – DINNOV-Enterprise –showed relative stability over the five years under analysis, while the contribution of those enterprises to employment and value added presented a declining trajectory. While still displaying a much higher average growth rate than the other economies included in this analysis, the Brazilian economy went through a progressive loss of dynamism during that period, and the latter indicators seemed to respond faster to the change in the macroeconomic environment than the first.

5. Conclusions

This article has shown the importance an indicator of innovation output can have to gauge the health of a national system of innovation and for ST&I policymaking and evaluation. Such an indicator can be useful as a tool to balance processes that are traditionally biased toward R&D policy instruments and indicators. Setting policy targets in terms of innovation output, in a manner similar to today’s targets measured in terms of R&D expenditures as percentages of GDP, would constitute a step further in such a balancing effort.

The EU 2020 indicator developed to fill that gap was described and assessed in section 3 of this article. It is a composite indicator based on component indicators that are not clear-cut indicators of innovation output. Furthermore, they rely on sectoral classifications that are applied as fixed templates for all countries at all times and are essentially insensitive to differences between countries’ innovation processes other than those that can be captured by changes in their sectoral structures. Surprisingly, data produced by innovation surveys are not used for the computation of the EU 2020 innovation output indicator.

Moreover, the complex process used to integrate the component indicators following sequential layers of transformations has unwanted consequences for the composite indicator. Tracing the impact of components on the composite indicator becomes a very difficult task. Most importantly, it also makes it very difficult i) to understand the indicator’s meaning and ii) to communicate its results to an audience that goes beyond the community of statistics producers. These are likely the most important reasons why the EU 2020 indicator seems to have failed to meet the expectations placed on it by the European Commission itself and by those who see an innovation output indicator as an important tool to help steer ST&I policies toward genuine innovation policies.

DINNOV, a new family of indicators that aims at capturing the output of innovative and high growth enterprises, was introduced in section 4. These indicators are measurements of the contribution dynamic and innovative enterprises give to the total number of enterprises, employment and value-added in the economy. They are computed at the enterprise level and do not rely on sectoral classifications, but they require access to microdata. The way they are computed avoids most of the drawbacks of the EU’s innovation output indicator, and they are much easier to communicate and to be understood by policymakers and the public in general.

Brazilian data were used to estimate the DINNOV indicator values for a few years. A proxy of the DINNOV-Enterprise indicator was also computed for Brazil and 17 European countries for the period 2008-2014. This exercise should be interpreted mainly as a proof of concept of the new indicators and as evidence that it is possible to compute them. However, much more is required for their validation. Further research must be undertaken. It is necessary to estimate them for a longer period of time, shorter intervals and many more economies.¹¹

Would it be possible to design and estimate innovation output indicators that go beyond the binary measurement of innovation and take into consideration the intensity of innovation or the importance of innovation for enterprise growth? Is the characteristic of being high growth and innovative a stable or persistent feature of enterprises that makes them good candidates for becoming policy targets? These are some of the questions that could guide further research to forge ahead in the quest for a genuine and good innovation output indicator. This article hopes to represent a small step in this direction.

ANNEX Rates of innovation and rates of high-growth-enterprises of selected countries

TABLE A1
Innovation rates, selected countries, 2008-2014 (%)

Country	2008		2009		2010		2011		2012		2013		2014		2008-2014 (avg.)
Brazil	38.4	(a)	37.1	(e)	37.1	(e)	35.9	(a)	36.1	(e)	36.1	(e)	36.3	(a)	36.7
Czechia	40.9	(a)	37.9	(e)	36.4	(a)	36.9	(e)	37.5	(a)	38.0	(e)	38.5	(a)	38.0
Denmark	40.8	(a)	42.4	(e)	44.7	(a)	41.2	(e)	38.0	(a)	37.9	(e)	37.9	(a)	40.4
Estonia	50.5	(a)	49.2	(e)	48.7	(a)	43.9	(e)	39.7	(a)	27.5	(e)	19.2	(a)	39.8
France	39.0	(a)	38.4	(e)	37.9	(a)	38.4	(e)	39.1	(a)	40.8	(e)	42.6	(a)	39.5
Hungary	19.4	(a)	17.7	(e)	16.6	(a)	15.8	(e)	15.1	(a)	16.2	(e)	17.3	(a)	16.9
Italy	41.5	(a)	42.5	(e)	43.9	(a)	43.6	(e)	43.6	(a)	40.2	(e)	37.4	(a)	41.8
Latvia	26.2	(a)	21.3	(e)	17.8	(a)	18.5	(e)	19.6	(a)	17.9	(e)	16.4	(a)	19.7
Lithuania	24.0	(a)	20.8	(e)	20.2	(a)	19.8	(e)	19.7	(a)	26.8	(e)	37.6	(a)	24.1
Luxembourg	44.5	(a)	47.1	(e)	51.0	(a)	50.3	(e)	52.1	(a)	47.7	(e)	44.2	(a)	48.1
Netherlands	39.7	(a)	45.2	(e)	51.5	(a)	48.8	(e)	46.4	(a)	47.9	(e)	49.5	(a)	47.0
Portugal	46.3	(a)	44.5	(e)	42.8	(a)	40.1	(e)	37.6	(a)	40.6	(e)	43.8	(a)	42.2
Romania	21.3	(a)	18.3	(e)	16.0	(a)	10.3	(e)	7.0	(a)	6.3	(e)	6.1	(a)	12.2
Slovakia	25.7	(a)	28.0	(e)	31.0	(a)	24.5	(e)	19.4	(a)	20.1	(e)	21.1	(a)	24.3
Slovenia	40.9	(d)	40.2	(d)	40.2	(d)	38.7	(d)	38.0	(d)	38.4	(d)	37.3	(a)	39.1
Spain	31.6	(a)	31.1	(e)	30.7	(a)	27.0	(e)	24.0	(a)	23.9	(e)	23.9	(a)	27.5
Sweden	46.9	(a)	47.1	(e)	47.5	(a)	45.7	(e)	44.1	(a)	43.7	(d)	43.3	(d)	45.5
United Kingdom	35.2	(b)	35.2	(b)	35.2	(a)	34.4	(e)	33.6	(a)	36.3	(e)	39.3	(a)	35.6

• Sources: Eurostat, Community Innovation Surveys - CIS <https://ec.europa.eu/eurostat/web/science-technology-innovation/data/database>, accessed on 09/03/2020. IBGE, Pesquisa de Inovação - PINTEC <https://sidra.ibge.gov.br/tabela/5453>, accessed 04/27/2020 (Authors’ elaboration).
• Note: Innovation rate here refers to the number of product and/or process innovative enterprises as a proportion of the total number of enterprises (10 or more employees) of the manufacturing activities.
• Legend: (a) Original data. (b) Assume the value of the first available year. (d) Calculated data. (e) Geometric average of the closest available years.

TABLE A2
Rates of high-growth enterprises, selected countries, 2008-2014 (%)

Country	2008		2009		2010		2011		2012		2013		2014		2008-2014 (avg.)
Brazil	8.0	(a)	7.2	(a)	7.3	(a)	7.0	(a)	6.9	(a)	6.1	(a)	5.6	(a)	6.9
Czechia	4.9	(a)	3.4	(a)	2.9	(a)	3.0	(a)	4.5	(a)	4.6	(a)	4.6	(c)	4.0
Denmark	1.9	(b)	1.9	(a)	1.5	(a)	1.2	(a)	1.3	(a)	1.3	(c)	1.3	(c)	1.5
Estonia	4.9	(a)	3.2	(a)	2.5	(a)	2.9	(a)	3.8	(a)	3.8	(a)	3.8	(c)	3.6
France	5.6	(b)	5.6	(a)	3.6	(a)	3.3	(a)	3.5	(a)	3.5	(c)	3.5	(c)	4.1
Hungary	4.4	(a)	3.1	(a)	3.0	(a)	3.5	(a)	4.7	(a)	4.2	(a)	4.2	(c)	3.9
Italy	2.1	(a)	1.8	(a)	1.4	(a)	1.4	(a)	1.3	(a)	1.4	(a)	1.4	(c)	1.5
Latvia	6.9	(b)	6.9	(b)	6.9	(b)	6.9	(a)	11.1	(a)	9.9	(a)	9.9	(c)	8.4
Lithuania	2.2	(b)	2.2	(b)	2.2	(a)	2.5	(a)	3.8	(a)	3.8	(a)	3.8	(c)	2.9
Luxembourg	4.0	(a)	1.6	(a)	1.9	(a)	2.2	(a)	2.2	(a)	2.2	(c)	2.2	(c)	2.3
Netherlands	4.2	(b)	4.2	(b)	4.2	(b)	4.2	(b)	4.2	(a)	3.9	(a)	3.9	(c)	4.1
Portugal	3.4	(a)	2.2	(a)	1.8	(a)	1.8	(a)	1.9	(a)	1.9	(a)	1.9	(c)	2.1
Romania	1.0	(a)	0.4	(a)	0.9	(a)	0.5	(a)	0.8	(a)	0.8	(a)	0.8	(c)	0.7
Slovakia	10.1	(b)	10.1	(b)	10.1	(b)	10.1	(b)	10.1	(a)	13.1	(a)	13.1	(c)	11.0
Slovenia	3.9	(a)	1.9	(a)	1.5	(a)	1.6	(a)	1.6	(a)	1.6	(c)	1.6	(c)	2.0
Spain	1.5	(b)	1.5	(a)	1.3	(a)	1.8	(a)	2.2	(a)	2.3	(a)	2.3	(c)	1.8
Sweden	3.3	(a)	3.3	(a)	3.3	(c)	3.3	(c)	3.3	(c)	3.3	(c)	3.3	(c)	3.3
United Kingdom	3.5	(b)	3.5	(b)	3.5	(b)	3.5	(b)	3.5	(a)	3.5	(a)	3.5	(c)	3.5

• Sources: OECD. Stat, “SDBS Business Demography Indicators” <https://stats.oecd.org/Index.aspx?DataSetCode=SDBS_BDI_ISIC4#>, accessed 03/03/2020. “Demografia das Empresas e Estatísticas de Empreendedorismo” (SIDRA, Tables 2719 and 1936). Accessed 16/12/2020. (Authors’ elaboration.)
• Note: High growth rate here refers to the number of high-growth enterprises as a proportion to the total number of enterprises. High-growth enterprises are those (of 10 or more employees) with average annualized growth greater than 20% per annum over a three-year period, where growth is measured by the number of employees. Manufacturing activities only.
• Legend: (a) Original data. (b) Assume the value of the first available year. (c) Assume the value of the last available year.

Acknowledgements

The first author wrote most of this article during his stay as visiting scholar at the Department of Economics of Hofstra University, Spring 2019, while he was enjoying a leave of absence from his position at the Brazilian Federal Senate. He is grateful for the support of both institutions. The article is based on a report of a research project previously developed by the authors of this paper (Viotti et al., 2015). The authors would like to acknowledge the support received for the development of the original report from the Brazilian Center for Strategic Studies and Management (CGEE) and the Brazilian Agency for Industrial Development (ABDI). They would also like to acknowledge the Brazilian Institute of Geography and Statistics (IBGE) for giving access to databases necessary for the development of the research project. The authors alone are responsible for this article and the opinions and arguments employed herein do not necessarily reflect the views of the institutions or persons that directly or indirectly supported its elaboration.

• 1
  According to Scopus statistics compiled by the Brazilian Ministry of ST&I (MCTI). <https://antigo.mctic.gov.br/mctic/opencms/indicadores/detalhe/Producao_Cientifica/Producao_Cientifica_5.5.html> Accessed 12/10/2020.
• 2
  According to estimates based on USPTO data <https://www.uspto.gov/web/offices/ac/ido/oeip/taf/st_co_96.htm> and <https://www.uspto.gov/web/offices/ac/ido/oeip/taf/st_co_19.htm> Accessed 12/10/2020.
• 3
  The authors thank an anonymous referee for calling their attention to this phenomenon and to the fact that the UN Committee of Experts on Business and Trade Statistics is currently debating alternative approaches to deal with it.
• 4
  A country’s score in a given year, for instance, cannot be properly compared to the score it got in the previous year because it depends not exclusively on each country’s innovative performance, it is also influenced by the distribution of the scores of all countries in the same year due to the renormalization process the scores are submitted to.
• 5
  It is interesting to recall that the guidelines set for the creation of the innovation output indicator by the high-level panel established that the first desirable property of the indicator is to be “simple and understandable”, and that this criterion would “exclude a composite indicator” (EUROPEAN COMMISSION, 2010c, p. 5).
• 6
  Interestingly, a large review of a different strain of literature (BOTELHO et al., 2021) concluded, “recent research provides evidence that the relationship between entrepreneurship and economic growth is driven (…) by a small subset of high-growth startups that are primarily categorized as innovation-driven.”
• 7
  The third edition of the Oslo Manual (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 2005) was the methodological reference for all the Brazilian and CIS surveys of innovation used in the DINNOV estimations. The fourth edition of the manual came out in 2018 (ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, 2018) with important improvements, but up to now no Brazilian survey and just one round of the CIS was conducted according to the new guidelines, and their results are not comparable with those of the previous surveys. Despite the observed changes, however, and taking care to use longitudinal comparable data, we see no impediment for the future computation of DINNOV indicators using information from surveys based on the Oslo Manual´s newest edition.
• 8
  The microdata accessed by this research project were limited to a small number of years due to reasons beyond the authors’ control, among which the postponement of the last Brazilian survey (PINTEC 2017, concluded in 2020), and the Coronavirus pandemic. In spite of that, the exercise was deemed valid for the purpose of generating original estimations of the innovation output indicators introduced by this article, and specially for serving as a proof of concept of those indicators.
• 9
  These indicators were estimated for firms with 10 or more employees and for the economic activities included in the scope of PINTEC, which are Mining and quarrying; Manufacturing; and selected services from Information and Communication; and Professional, scientific and technical activities.
• 10
  The values of C/B for the years 2009 and 2010 in all three tables were obtained by an interpolation between their values for the years 2008 and 2011, while its value for the year 2012 was obtained by an extrapolation of its value for 2011.
• 11
  EUROSTAT recognizes incidentally that “attempts for a concrete measurement of the outcome of business innovation in the CIS have not yet been fully convincing”, but the institution renewed its commitment “to complement the CIS information with key variables produced in the Structural Business Statistics by using MDL [microdata linking] techniques” (EUROSTAT, 2021, p. 8), and this effort should enable the estimation of DINNOV indicators for EU economies in the near future. On the other hand, the new IBGE’s commitment to add biannual small innovation surveys to the regular Brazilian surveys, realized every three years, will create conditions for replacing the rough techniques of DINNOV indicators estimations for years between surveys proposed in this article <https://agenciadenoticias.ibge.gov.br/agencia-noticias/2012-agencia-de-noticias/noticias/30465-ibge-e-abdi-lancam-pesquisa-de-inovacao-semestral>.
• Source of funding: The development of this article did not receive direct funding. It is, however, an expansion of a research project supported by the Brazilian Center for Strategic Studies and Management (CGEE) and the Brazilian Agency for Industrial Development (ABDI).

References

• BOTELHO, T. L.; FEHDER, D.; HOCHBERG, Y. Innovation-driven entrepreneurship Cambridge: National Bureau of Economic Research, 2021. (Working Paper n. 28990)
• BRASIL. Ministério da Ciência, Tecnologia e Inovações. Estratégia nacional de ciência, tecnologia e inovação 2012-2015 Brasília, DF: MCTI, 2012. Available from: <https://livroaberto.ibict.br/218981.pdf>. Access in: 17 June 2021.
  » https://livroaberto.ibict.br/218981.pdf
• BRASIL. Ministério da Ciência, Tecnologia e Inovações. Estratégia nacional de ciência, tecnologia e inovação 2016-2022 Brasília, DF: MCTI, 2016. Available from: <https://antigo.mctic.gov.br/mctic/export/sites/institucional/ciencia/SEPED/Arquivos/PlanosDeAcao/PACTI_Sumario_executivo_Web.pdf>. Access in: 17 June 2021
  » https://antigo.mctic.gov.br/mctic/export/sites/institucional/ciencia/SEPED/Arquivos/PlanosDeAcao/PACTI_Sumario_executivo_Web.pdf
• CARVALHO, A. Wishful thinking about R&D policy targets: what governments promise and what they actually deliver. Science & Public Policy, UK, v. 45, n. 3, p. 373-391, 2018. http://dx.doi.org/10.1093/scipol/scx069
  » http://dx.doi.org/10.1093/scipol/scx069
• EDQUIST, C. Striving towards a holistic innovation policy in european countries - but linearity still prevails! STI Policy Review, Korea, v. 5, n. 2, p. 1-19, 2014.
• EUROPEAN COMMISSION. Developing an indicator of innovation output - Commission Staff Working Document Brussels: European Commission, 2013b. Available from: <https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013SC0325>. Access in: 04 Aug 2022.
  » https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013SC0325
• EUROPEAN COMMISSION. Elements for the setting-up of headline indicators for innovation in support of the Europe 2020 strategy - Report of the High-Level Panel on the Measurement of Innovation (September 30, 2010) Luxembourg: Publications Office of the European Union, 2010b.
• EUROPEAN COMMISSION. Europe 2020 flagship initiative innovation union, SEC (2010) 1161 Brussels: European Commission, 2010a. Available from: <https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52010DC0546>. Access in: 04 Aug 2022.
  » https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52010DC0546
• EUROPEAN COMMISSION. Eurostat-OECD Manual on business demography statistics -2007 2008. Available from: <http://www.oecd.org/sdd/business-stats/39974460.pdf>. Access in: 17 Apr 2019.
  » http://www.oecd.org/sdd/business-stats/39974460.pdf
• EUROPEAN COMMISSION. Measuring innovation output in Europe: towards a new indicator Brussels: European Commission, 2013a. Available from: <https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013DC0624>. Access in04 Aug 2022.
  » https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013DC0624
• EUROPEAN COMMISSION. New indicator of innovation output, MEMO/13/782 Brussels. Available from: <https://ec.europa.eu/commission/presscorner/detail/en/MEMO_13_782>. Access in: 04 Aug 2022. 2013c.
  » https://ec.europa.eu/commission/presscorner/detail/en/MEMO_13_782
• EUROSTAT. Community innovation survey: new features 2021. Available from: <https://ec.europa.eu/eurostat/statistics-explained/SEPDF/cache/101661.pdf>. Access in: 25 Mar 2022.
  » https://ec.europa.eu/eurostat/statistics-explained/SEPDF/cache/101661.pdf
• FREEMAN, C. Technology policy and economic performance: lessons from Japan London: Pinter, 1987.
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Estatísticas de empreendedorismo 2008 Rio de Janeiro: IBGE, 2011. Available from: <https://biblioteca.ibge.gov.br/visualizacao/monografias/GEBIS%20-%20RJ/empreendedorismo_2008.pdf>. Access in: 04 Aug 2022.
  » https://biblioteca.ibge.gov.br/visualizacao/monografias/GEBIS%20-%20RJ/empreendedorismo_2008.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Estatísticas de empreendedorismo 2010 Rio de Janeiro: IBGE, 2012. Available from: <https://biblioteca.ibge.gov.br/visualizacao/livros/liv61536.pdf>. Access in: 04 Aug 2022.
  » https://biblioteca.ibge.gov.br/visualizacao/livros/liv61536.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Estatísticas de empreendedorismo 2011 Rio de Janeiro: IBGE, 2013b. Available from: <http://www.ibge.gov.br/home/estatistica/economia/empreendedorismo/2011/default.shtm>. Access in: 22 Feb 2015.
  » http://www.ibge.gov.br/home/estatistica/economia/empreendedorismo/2011/default.shtm
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Estatísticas de empreendedorismo 2012 Rio de Janeiro: IBGE, 2014a. Available from: <http://ftp.ibge.gov.br/Estatisticas_de_Empreendedorismo/2012/empreendedorismo2012.pdf>. Access in: 16 Feb 2015.
  » http://ftp.ibge.gov.br/Estatisticas_de_Empreendedorismo/2012/empreendedorismo2012.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Pesquisa de inovação tecnológica 2008 Rio de Janeiro: IBGE, 2010. Available from: <http://www.pintec.ibge.gov.br/downloads/PUBLICACAO/Publicacao%20PINTEC%202008.pdf >. Access in: 11 Apr 2010.
  » http://www.pintec.ibge.gov.br/downloads/PUBLICACAO/Publicacao%20PINTEC%202008.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Pesquisa de inovação 2011 Rio de Janeiro: IBGE, 2013a. Available from: <http://ftp.ibge.gov.br/Industrias_Extrativas_e_de_Transformacao/Pesquisa_de_Inovacao_Tecnologica/2011/pintec2011.pdf>. Access in: 22 Feb 2015.
  » http://ftp.ibge.gov.br/Industrias_Extrativas_e_de_Transformacao/Pesquisa_de_Inovacao_Tecnologica/2011/pintec2011.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Pesquisa industrial 2011 Rio de Janeiro: IBGE, 2013c. Available from: <https://biblioteca.ibge.gov.br/visualizacao/periodicos/1719/pia_2011_v30_n1_empresa.pdf>. Access in: 22 Feb 2015.
  » https://biblioteca.ibge.gov.br/visualizacao/periodicos/1719/pia_2011_v30_n1_empresa.pdf
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA. Pesquisa industrial 2012 Rio de Janeiro: IBGE, 2014b. Available from: <https://biblioteca.ibge.gov.br/visualizacao/periodicos/1719/pia_2012_v31_n1_empresa.pdf>. Access in: 04 Aug 2022.
  » https://biblioteca.ibge.gov.br/visualizacao/periodicos/1719/pia_2012_v31_n1_empresa.pdf
• JANGER, J. et al The EU 2020 innovation indicator: a step forward in measuring innovation outputs and outcomes? Research Policy, Japan, v. 46, p. 30-42, 2017.
• LUNDVALL, B. Å. Innovation as an interactive process: from user-producer interaction to the National System of Innovation. In: DOSI, G. et al. (Eds.). Technical change and economic theory London: Pinter, 1988. p. 349-369.
• LUNDVALL, B. Å. National systems of innovation: towards a theory of innovation and interactive learning. London: Pinter, 1992.
• NELSON, R. R. Institutions supporting technical change in the United States. In DOSI, G. et al. (eds), Technical change and economic theory London, Pinter, 1988. p. 312-329.
• NELSON, R. R., editor. National innovation systems: a comparative study. Oxford: Oxford University Press, 1993.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. Dynamizing national innovation systems Paris: OECD Publishing, 2002.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. EUROSTAT. Oslo Manual 2005: guidelines for collecting and interpreting innovation data. Paris: OECD Publishing, 2005. 3rd ed. https://doi.org/10.1787/9789264013100-en.
  » https://doi.org/10.1787/9789264013100-en.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. EUROSTAT. Oslo Manual 2018: guidelines for collecting, reporting and using data on innovation. Paris: OECD Publishing, 2018. 4th ed. https://doi.org/10.1787/9789264304604-en
  » https://doi.org/10.1787/9789264304604-en
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. Managing national innovation systems Paris: OECD Publishing, 1999.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. National innovation systems. Paris: OECD Publishing, 1997.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. OECD Patent Statistics Manual Paris: OECD Publishing, 2009. Available from: <http://www.oecd-ilibrary.org/science-and-technology/oecd-patent-statistics-manual_9789264056442-en>. Access in: 27 Mar 2014.
  » http://www.oecd-ilibrary.org/science-and-technology/oecd-patent-statistics-manual_9789264056442-en
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. OECD Proposed guidelines for collecting and interpreting technological innovation data – Oslo Manual. Paris: OECD Publishing, 1992. (OECD/GD(92)26). Available from: <http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=OCDE/GD(92)26&docLanguage=En>. Access in: 15 Mar 2019.
  » http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=OCDE/GD(92)26&docLanguage=En»
• STOKES, D. E. Pasteur’s quadrant: basic science and technological innovation. Washington: Brookings Institution Press, 1997.
• UNITED NATIONS CONFERENCE ON TRADE AND DEVELOPMENT. A framework for science, technology and innovation policy reviews: helping countries leverage knowledge and innovation for development. Geneva: UNCTAD, 2011.
• VÉRTESY, D. The innovation output indicator 2017: methodology report. Luxembourg: Publications Office of the European Union, 2017. (JRC Technical Report, JRC108942, EUR 28876).
• VÉRTESY, D.; DEISS, R. The innovation output indicator 2016: methodology update. Luxembourg: Publications Office of the European Union, 2016. (JRC Technical Reports, EUR 27880).
• VIOTTI, E. B. Brasil: de política de C&T para política de inovação? Evolução e desafios das políticas brasileiras de ciência, tecnologia e inovação. In: VELHO, L.; SOUSA-PAULA, M. C. (Org.). Avaliação de políticas de ciência, tecnologia e inovação: diálogo entre experiências internacionais e brasileiras Brasília: Centro de Gestão e Estudos Estratégicos (CGEE), 2008a. p. 137-174. Available from: <https://www.cgee.org.br/documents/10195/734063/19Avalia%C3%A7%C3%A3o+CTeI%5B1%5D_6420.pdf/4eec2c77-38af-4be1-b001-7064cad5a0c4?version=1.3>. Access in: 21 Mar 2019.
  » https://www.cgee.org.br/documents/10195/734063/19Avalia%C3%A7%C3%A3o+CTeI%5B1%5D_6420.pdf/4eec2c77-38af-4be1-b001-7064cad5a0c4?version=1.3
• VIOTTI, E. B. Brazil: from S&T to innovation policy? The evolution and the challenges facing Brazilian policies for science, technology and innovation. In: THE 6TH INTERNATIONAL CONFERENCE OF GLOBELICS, 2008, Mexico. Proceedings… Atlanta: Georgia Tech Library, 2008b. [English version of VIOTTI (2008a)]. Available from: <http://hdl.handle.net/1853/36899>. Access in: 21 Mar 2019.
  » http://hdl.handle.net/1853/36899
• VIOTTI, E. B. et al Indicadores de resultado da inovação Brasília, DF: Centro de Gestão e Estudos Estratégicos (CGEE), 2015. Available from: <https://www.cgee.org.br/indicadores-de-resultado-da-inovacao>. Access in: 21 Mar 2019.
  » https://www.cgee.org.br/indicadores-de-resultado-da-inovacao
• VIOTTI, E. B. National Learning Systems: a new approach on technological change in late industrializing economies and evidences from the cases of Brazil and South Korea. Technological Forecasting and Social Change, v. 69, n. 7, p. 653-680, 2002. http://dx.doi.org/10.1016/S0040-1625(01)00167-6
  » http://dx.doi.org/10.1016/S0040-1625(01)00167-6
• VIOTTI, E. B. Um novo indicador de meta-síntese para a política de inovação. Parcerias Estratégicas, v. 18, n. 36, p. 151-174, 2013. Available from: <http://seer.cgee.org.br/index.php/parcerias_estrategicas/article/viewFile/715/656>. Access in: 04 Aug 2022.
  » http://seer.cgee.org.br/index.php/parcerias_estrategicas/article/viewFile/715/656

Publication Dates

• Publication in this collection
  22 Aug 2022
• Date of issue
  2022

History

• Received
  17 June 2021
• Reviewed
  28 June 2022
• Accepted
  04 July 2022