Concordance measurement applying the Delphi method: proposal and evaluation of a set determinants factor of the Triple Helix existence

Abstract

Paper aims

This article presents the development and results of research carried out to know the modus agendi of a Regional Triple Helix and measure the intensity of the presence of a set of factors proposed as determinants of its existence and consolidation.

Originality

This is an unprecedented contribution to research in TH, as it investigates how much an established regional reality matches the pattern of a modus agendi and a set of determining factors of its existence and consolidation.

Research method

The inductive basic research method was adopted, with qualitative and quantitative approach, exploratory and descriptive objective and field research, applying the e-Delphi method.

Main findings

A high degree of misalignment of perceptions was identified among respondents and the existence of a Triple Helix that is still consolidating.

Implications for theory and practice

The relevance of this article is corroborated by the unprecedented application of the calculation of three statistical indicators (CVI-h; CV; IR) in the e-Delphi method. Another important contribution of this research lies in the fact that it investigates the level of alignment of the modus agendi of a TH, with a set of determinant factors that indicate the existence and consolidation of this development.

Keywords
Triple helix; Delphi method; Concordance Validation Index; Concordance measurement; Enabling solutions; Innovation

1. Introduction

Academic research in the field of university-government-industry interactions has focused mainly on describing the roles assigned to each, and in general, considered the existence of these three institutions in a region, as an end condition, and not a means, for the establishment of the Triple Helix (TH) model.

It is noteworthy that through the consulted works that few studies carried out demonstrate immersion in the modus agendi of the institutions and their effective contributions to the construction of the synergy indispensable for the consolidation of the TH model.

When inferring through the works of Sum & Jessop (2013) that constructs in the TH area are restricted to investigative studies of implantations and occasional successful cases, this article aims to instigate a community on the importance of the daily practice of the integration between the propellers of the TH, as essence for its functioning and effectiveness.

Researches in the Web databases (Science Direct, Scopus and Web of Science) demonstrate the growing interest of researchers in relation to the TH model (Etzkowitz & Zhou, 2017; Holland & Oliveira, 2014). This model regionally designed for the field of innovations, in recent years has gained notoriety in academic approaches to development and regional sustainable competitive advantage.

However, despite being experienced through implantation in several locations, studies demonstrate instabilities, setbacks and distortions caused, a priori, by the lack of adequacy to the guiding assumptions of the effectiveness and solidity of the TH model prescribed by Etzkowitz & Zhou (2017).

The verticalized governance style from the top tends to build impasse in practice when it is not possible to reach consensus among stakeholders. In this context, participatory approaches are needed, focusing more on the cultivation of social capital and a new economy (Zhai & Ng, 2013).

It can be inferred that it is necessary to align the speeches of the representatives of the instituted powers, the economic agents and the other organizations that make up society, in the search for the self-development of communities in general. Concomitantly, perceived the scarcity of academic constructs based on the aspects of distortion, non-consolidation and disarticulation of the TH model, this work is expected to contribute to the gradual closing of this research gap.

Thus, the objective of this work is to identify and measure the predominance of consensus or dissent on the adequacy of the modus agendi of a TH, from a set of factors indicative of the consolidation of a TH in a given region.

This article is structured in four more sections. The second Section presents the theoretical framework containing the conceptual basis of the work, covering the Triple Helix Model and the Delphi method. The methodological path is presented in detail in Section 3, while Section 4 is intended to describe the stages of the e-Delphi research. The fifth Section presents the full panelists' responses to the propositions presented and the research conclusions. At the end of the article are the bibliographic references consulted in the preparation of this text.

2. Background review

Observing the results of the largest world economies, it is identified that the integration and cooperation among the members of the TH model, aligned with the common objective of economic development and strengthening, has been a preponderant factor for the growth and hegemony of the select group of more developed countries.

The approach, the integration and the cooperation between the university, industry and the government are contributing factors to innovation and growth in a knowledge-based economy (Etzkowitz, 2013). For Chung & Park (2014) the growing importance of the model has contributed to the enrichment of theoretical and empirical research.

This triple alliance consists, among other aspects, in the provision of human capital by the university, in the regulatory role assumed by the government and in the application of new technologies in the production of goods and services by the industry (Lee and Kim, 2016).

The triple helix model requires reciprocity and alignment of its members' purposes, and when well implemented it immerses the important role played by universities, little considered by predecessor models (Verlinde & Macharis, 2016).

By observing the role of small municipalities dedicated to educational activity to generate opportunities, jobs and encourage entrepreneurial activity, this work brings an investigative look at a TH, in relation to a set of determining factors for the recognition of the consolidation of this model inserted in Brazilian Electronic Valley.

3. The Brazilian Electronic Valley

The option for the Brazilian Electronic Valley as a locus of research is justified by its expressiveness on the national scenery, as a field for establishing local productive arrangements. The Brazilian Electronic Valley, located in the southeast region of Brazil, has 153 companies, 89 of which are active in the electronics sector, 90% are considered micro-enterprises and 8% are medium-sized companies, which employ 29% of the specialized workforce in the electronic sector in the state. Annual revenue from the manufacture of more than 14,000 items exceeds US$600 million.

The development activities are linked to the teaching of Electronics, Telecommunications, Information Technology, and Administration, with the main characteristics of entrepreneurship and the encouragement of technological innovation. Its products are mainly focused on the following sectors: Electronics, Telecommunications, Security, Information Technology, Broadcasting, Industrial Automation, Building, Information Technology, Supplies, and Services.

The training and qualification of local professionals are promoted by pioneering institutions such as the Francisco Moreira da Costa Electronic Technical School (ETE), the first in Latin America (1959), the National Telecommunications Institute (INATEL, 1965), the Higher Education Center, Management, Technology, and Education - FAI (1971) and SESI / SENAI (2002) - the institution that promotes education and innovation for the industrial sector in the region.

In this sense, an incipient interaction between the governmental, business, and academic means can be seen in the communities that make up the Brazilian Electronic Valley. Committed to local and regional development, support, and development institutions participate through capital injection, project financing, economic feasibility analysis studies, consulting and information, development, and innovation, theoretical study, and entrepreneurship practices.

The local Triple Helix has the participation of around 30 institutions distributed in the activities of training the workforce, business incubators, industrial and commercial associations, entrepreneurship centers, public and private financing agencies.

Among the special highlights is the ICC - Inatel Competence Center, which offers a variety of services and technological solutions to the market, working as an extension of the R&D center of its customers and partners, enabling them to innovate in their products and/or processes and the specialized training of its employees and customers.

Another highlight at the Brazilian Electronic Valley is the annual holding of several fairs with national and international projections, promoted by union entities, technical schools, and colleges whose purpose is to show the work developed by entrepreneurial students and to promote products and services from the companies present in the arrangement production site.

4. The Triple Helix model – literature review

It is noted that an economy must consider the existence of identity between political, educational and socio-productive agents, as a way of creating bases capable of ensuring the stability of the organizations and communities in which they are established.

As asserted by Oliveira (2012), the development of the industry depends on the active participation of the educational institutions, whose researchers can guide the industrial environment, in shortening the distances between the conception of ideas and the effective creation of new products and services with greater added value, aiming to increase the national competitive advantage, vis-à-vis the external economies (Oliveira, 2012).

When realizing that the development built along the lines of the Triple Helix, due to its dynamism characteristics, can contribute to innovation, the relationships established in the three spheres - university - government - industry are identified, as the basis for sustaining the generation of new knowledge that culminate in enabling solutions in the field of innovations (Manzini, 2014).

In this sense, Oliveira (2012) indicate that the teaching and practice of entrepreneurship in universities and business incubators, when interconnected are able to create favorable conditions for regional socioeconomic development (Oliveira, 2012).

As an interconnection way, from the approach of Eberhart & Pascuci (2014), it can be inferred that an integration model designed according to a triple helix involves cooperation processes, in particular enabling the integration of government and companies in a process whose presence of universities is configured as essential for social and economic development.

Related to online inter-communication field Kobza & Mutlucan (2016, p. 291), emphasized that “[...] social networks can provide not only resources and support but also new and holistic perspectives, thanks to different perspectives and contributions from members”. And they add “[...] today, the entrepreneurial mindset is often derived from young professionals and students. Also, this mindset can be multiplied by the impact of your social networks”.

It is identified through the work of Kapetaniou & Lee (2017) that the current role of universities in the knowledge-based economy can be analyzed using the TH model. This approach, which emphasizes the critical role of interactions between universities and other actors, conceptualizes and highlights the current role of universities in the innovation process.

It is noteworthy that urban and regional developments demand looks, initiatives, and solutions that contribute to improving the associations of individuals while providing these conditions for social and intellectual development and better standards of quality of life.

According to Etzkowitz & Zhou (2017),

The triple helix is a universal model of innovation. It is the secret behind the development of Silicon Valley through sustainable innovation and entrepreneurship. The ‘organized’ Triple Helix is an invisible institutional tool for regional economic growth and social development. (Etzkowitz & Zhou, 2017, p. 25-30).

Still in this sense the authors highlights,

The University is the fundamental institution of knowledge-based societies, just as government and industry were the main institutions of industrial society. The industry continues to be a protagonist in the scope of production and the government is still the source of contractual relations that guarantee stable interactions and exchanges. (Etzkowitz and Zhou, 2017, p. 31).

Vlaisavljevic et al. (2020) discuss the well-structured nuclei in the triple helix model as representatives of the various departments of industries, incubated companies, researchers, and professionals linked to the sectors that support the development of governments, who work in complementary collaborative activities, create, disseminate and apply new technologies in the development of innovative products and services.

When addressing aspects related to innovation and entrepreneurship in the Triple Helix model, Etzkowitz & Zhou (2017) asserted that a regional Triple Helix for innovation and entrepreneurship must be created based on six principles showed in Table 1:

Table 1
Principles for creating a Triple Helix.

1	The University needs to represent a fundamental driver of a knowledge-based economy. It is an important piece to develop spaces for knowledge, innovation, and consensus.
2	The formation and development of knowledge-based companies are the results of interactions between the primary and secondary actors of a Triple Helix.
3	The government's role in Triple Helix should be moderator and its goal is to ensure that Triple Helix works well. The government may be the best candidate to create a “consensus space” that brings together the relevant actors to design and implement innovation projects.
4	Venture capital can act as a company or as an arm of a company, government, university, or foundation and is an important driver for the formation and growth of companies.
5	Innovation activities take place mainly in various organizations resulting from the intellectual effort of an “innovative entity”, and not from a single inventor. Incubators, accelerators, and technology transfer offices promote startups and innovative development in a given region, supported by city halls, universities, and sectorial business associations, among others.
6	Innovation should be perceived as an endless process, considering that Triple Helix, as a model for maintaining and developing the process, is a universal theory of innovation and entrepreneurship

• Source: Etzkowitz & Zhou (2017, p.33).

In line with the principles suggested by Etzkowitz & Zhou (2017), Oliveira et al. (2018), asserted that the TH model is perceived as a set of initiatives, commitments and effective actions promoted by teams representing

[...] communities, organizations, educational institutions and government, aligned with socioeconomic development, progress and continued existence of communities where they are established. (Oliveira et al., 2018, p. 1).

It is added that the integration of this set of organization is perceived as fundamental for the proper functioning of a model of TH, as shown in Figure 1.

Figure 1
Triple helix integrative rotation.

The form originally proposed by Leydesdorff (2013) and Etzkowitz (2017) for TH considered the existence of an interface area between the three agents, Government - University - Industry in an evaluative perspective of the efforts to promote changes in the relevant interfaces of science-technology-industry.

In this sense, rethinking the continuity and growth of these communities becomes an internal exercise for its citizens. In this scenario, there is an urgent need to strengthen collective thinking and acting in the face of the demands of the changing world economy.

Through the approach of Lawthon & Leydesdorff (2014), the TH model can be considered as an empirical heuristic that determines not only the economic forces, legislation and regulation by governments (regional or national), but also the endogenous dynamics of transformations by science-based inventions and innovations.

With regard to Brazil, Pellin & Engelmann (2018) highlight that the distancing from government-industry-university institutions leads to a condition of low systemic maturity, mainly due to the imposition of traditionalist bases of its historical-cultural process perceived as lethargic in response to the competitive world economic scenario, and by the ties to the state institutionality that distances the country from modernity.

It is worth mentioning that urban and regional development requires views, initiatives and solutions shared by organizations and individuals with the capacity to suppress dissent, and to act proactively in favor of building consensus aimed at collective objectives.

5. The Delphi method – literature review

The Delphi Method was developed by RAND Corporation in the 1950s, originally to predict the impact of technology on war. The method involves a group of experts who respond anonymously to questionnaires and later receive feedback in the form of a statistical representation of the “group response”, after which the process is repeated to arrive at something closer to the experts' consensus (Landeta et al., 2011).

It is a method used to examine a complex problem through a group of experts. The specialists generate a data source due to their knowledge and experience in relation to the subject under investigation.

In contrast to the commonly used method of brainstorming in which experts convene in one place to reach a consensus the Delphi method, allow to the experts to respond anonymously (Kim et al., 2013). Data are collected through questionnaires during the application of successive rounds interspersed by various iterations between researcher and respondents (Strasser, 2017).

The method has been used for forecasting scenarios, prioritizing problems, selecting topics, and defining research questions. Studies with application of the Delphi Method, in which the questions presented to specialists are more comprehensive, the contributions of the panelists may converge to more specific questions in relation to the subject.

Giannarou & Zervas (2014, p. 77) assert

Delphi is useful in case study analyses, because of its limitation of non-generalizability of the results, and provides a great advantage for the researcher who does not need a representative sample to implement this method.

Linstone & Turoff (2011) pointed out that some systems of discussion panels began to add voting options that would present a more integrated process for conducting a survey using the Delphi method.

The authors also highlighted that “However, the growing number of ways to conduct Delphi research on the Web has definitely led to the beginning of a new growth curve in the use of this method to support large collaborative groups.” (Linstone & Turoff, 2011).

Works that are directed to the development of theory can be enriched with the application of the Delphi Method, for greater understanding of the context and generation of hypotheses. It should be noted that e-Delphi represents an economical and practical way to use consensus-building strategies without the distance restrictions imposed by geography.

Group research often requires significant investments in data management. Tracking personal information and managing data collection and analysis are common challenges. In both qualitative and quantitative research, there is a growing use of electronic data management software, whose advantages include ease of data management, reduced risk of errors and greater transparency in the analytical process.

De Carli et al. (2010) stated

This method is applied as a research tool when there is no complete knowledge about a problem or phenomenon, and it works especially well when the objective is to improve the understanding of problems, opportunities, solutions, or to develop predictions. (De Carli et al., 2010, p. 553).

In the booming era of cloud computing, researchers must strongly consider the important security and access benefits that research efforts, such as e-Delphi, allow. Such benefits for managing research data will certainly proliferate over time; therefore, researchers should consider adopting the new web- technologies when designing and conducting e-Delphi studies.

In view of the evidence of these advantages, a bibliographic search was carried out to identify the extent of the academic investigation already done applying e-Delphi research. As noted during the literature review, the growing interest in use of the e-Delphi (versus traditional Delphi) it is justified by evidence of its benefits.

Delphi Method was conceived as a group communication process, and it aims, through successive rounds, to achieve a convergence of opinion on a specific issue in the real world. It is a method that uses a scientific technique that allows analyzing qualitative and quantitative data to identify and quantify the opinions of experts called panelists. It is understood that there is an agreement on the four fundamental pillars of the technique, namely: anonymity, the use of specialists, the application of interactive and feedback rounds, and the search for consensus or dissent (Giannarou & Zervas, 2014).

The method is based on a process of consulting opinions in order to allow individuals to explore and contribute to the solution of a complex problem. For Cortez & Johnston (2017), the Delphi Method is an iterative and structured process used to elicit, collect and aggregate opinions and judgments in a context of collective decision. The method can be used when there is incomplete knowledge about the phenomena.

Through the realization of a series of questionnaires, specific propositions are presented to the participants. The aggregated results are then delivered to the requesting researcher in order to, if necessary, reformulate the presented propositions.

The number of rounds elaborated varies according to the degree of consensus reached by the specialists, this consensus being understood at the individual level, that is, if there is a very high discrepancy in the opinion of specialists in the various rounds, will not possible to reach a consensus. It should be noted that the specialist may change his opinion in relation to the questions he considers most relevant, once new questions are introduced in the questionnaires, when successive rounds are applied.

Delphi Method has a specific terminology, which defines each of its activities, stages, as well as the subjects involved.

Facilitator - Person responsible for preparing the questionnaires, sending them to specialists, collecting and analyzing the responses;

Panelist - Person whose education and / or professional experience entitles them to give an opinion on the subject of the research;

Panel - It is the group of specialists who will act as panelists;

Questionary - It is the document (instrument) that is sent to the specialists. It is not only a document that contains a list of questions, but a document with which research participants are able to interact, since the results of previous circulations are presented in it;

Round - It is each of the successive questionnaires presented to the group of experts.

During the development of the research using the e-Delphi method, the facilitator also accumulates, as a fundamental activity, the establishment of periodic contacts with the panelists, in order to seek their commitment to the deadlines required for the completion of the research, according to the previously established schedule (Linstone et al., 2011; Ameyaw et al., 2016).

6. Research method

This chapter aims to describe the methodological guidelines that guided this research, and the techniques and procedures for collecting and analyzing the data obtained to achieve the proposed objectives.

In this work, the inductive method of basic research was applied, adding a qualitative approach through bibliographic and documentary procedures and a quantitative approach with the development of a survey, as described in Figure 2. Using the e-Delphi technique a combined approach was carried out, applying qualitative and quantitative approaches.

Figure 2
Research methodological flow.

In this sense Ameyaw et al. (2016) emphasized, “Compared with traditional Delphi studies, quantitative Delphi studies require careful research design and consequently a number of statistical data analysis approaches.”

As shown in Table 2, regarding the type, the research was classified as exploratory and descriptive, considering the development of field research with the purpose of knowledge of the researched region and the modus agendi of the local TH. Bibliometric analysis was added to the technical procedures.

Table 2
Research classification.

Regarding the method		Inductive	Regarding procedures
As for nature		Applied
Regarding approaches		Qualitative	Bibliographic and documentary survey
		Quantitative	Survey
		Combined	Delphi method
As to type or modality		Exploratory	Field research
		Descriptive	Case study

• Elaborated by the authors.

Investigative work when carried out in the field and in the light of the collection of opinions from individuals, in addition to the research instruments demand a minimum set of hypotheses, perceived here as propositions that are admitted, regardless of whether it is true or false, as a principle from which a given set of consequences, assumptions and conjectures can be deduced as possibility considered valid before confirmation is made.

In this sense, the importance of the researcher's immersion is emphasized, particularly when conducting a field study (Woodside, 2010; Yin, 2017).

It should be noted that when opting in a similar way to this research, using the e-Delphi Method, it is recommended to the researcher, to consider that both consensus and dissent can result as an expected result at the end of the panelists argumentation process.

The option for the Delphi method of research in order to verify the levels of consensus and dissent demands the application of statistical techniques, observing the Interquartile Range as one of the applied measures (Linstone & Turoff (2011); Miguel (2012) and instruments for a structuring interactive group communication.

Creswell (2014) suggests that quantitative methods are suitable for deductive approaches, where a theory or hypothesis justifies variables, states purposes and indicates narrowly defined research questions. Researchers of mixed methods, usually organize the report of procedures in the collection of quantitative data and in the analysis of the content of qualitative information. Creswell adds,

[...] then, in the conclusions or in the interpretation phase of the study, the researcher measures how the qualitative information helped to elaborate or expand the quantitative results. (Creswell, 2014, p. 220).

The feature of interactivity is highlighted as essential to the process of forming consensus by the e-Delphi method, whose consolidation does not dispense with a systemic community, capable of sharing information and knowledge in a network, in which they must participate: local actors with scientific knowledge, representatives of the business community, public and private development agents and institutions representing society.

The feature of interactivity is highlighted as essential to the process of forming consensus by the e-Delphi method, whose consolidation does not dispense with a systemic community, capable of sharing information and knowledge in a network, in which they must participate: local actors with scientific knowledge, representatives of the business community, public and private development agents and institutions representing society.

Broadly speaking, society must be seen as the protagonist of the process, which must be horizontal, endogenous and emerging both in power relations and in the practice of individual and collective responsibilities.

Through field research, the delimited region was explored in order to describe the structure and status quo of the local TH. In this phase, records of the historical, cultural and economic evolution of the locus object of the research were also surveyed. In parallel, potential panelists for research development were identified using the e-Delphi Method via survey. The methodological flow through which it was proposed to build this research work is illustrated as shown in Figure 2.

The main required actions for the fulfillment of the outlined objectives are showed in the graphic representations of the research method and the methodological flow, and these actions, together, will lead to the fulfillment of the general objective of this work.

7. Research development - Choice of painelists

In his approach to the Delphi method, Grisham (2009) recommends selecting a panel of panelists that is balanced in terms of impartiality and interest in the subject. The term “panelist” means “one who has a notorious knowledge, in addition to proven experience and qualification, linked to academic and professional institutions that have an interface with the object of the study” Grisham (2009, p. 117). In general, the identification and choice of organizations and people who will constitute the group of panelists must follow a general guide as suggested in Figure 3.

Figure 3
Guide to constitute the panelist group.

Regarding the experts, Nogueira & Fuscaldi (2018) recommend the following characteristics:

1. a

   To have a cultural level compatible with the complexity of the subject;

2. b

   Be a studious of the subject;

3. c

   To have notorious knowledge in the area to be debated, in academic circles and in the areas of research, social and professional;

4. d

   Acting in related professional functions;

5. e

   Have experience, that is, the time he dedicates to the topic as a researcher, teacher or in public or private functions;

6. f

   Be author of publications evaluated according to the number and importance of the works published in relation to the researched theme;

7. g

   Have participated in national and international events that are related to the subject of the study.

The specialists were chosen by means of a non-probabilistic and intentional sampling, considering as an initial criterion the link with the institutions predefined in the TH Model such as representative bodies of the public power, technical and higher education institutions, companies and private sector organizations.

After an initial survey, a preliminary list of specialists was established based on employment or social link with government sectors, public and private companies, secondary and higher education institutions and representative bodies of classes. The refinement of the panelists' choice considered the content of the academic curriculum and publications in classified journals.

In all, twenty-five experts were invited to form the group of panelists, out of which, fifteen professionals agreed to participate in the research. They were divided into three groups:

1. a

   The Group 1 was formed by local academic researchers who had the following prerequisites:

• Experience in the creation and development of industrial clusters;

• Experience in implementing activities aimed at economic and social entrepreneurship;

• Professional experience in activities to promote the innovation of products and services.

• Business incubator management.

1. b

   The Group 2 was composed by, executives of local private companies operating in the electronics industry, a type of industry prevalent in the region and in the trade associations. The chosen professionals fulfilled the requirement of having more than 5 years of professional experience and experience in the researched region.

2. c

   The Group 3 was formed by professionals, with five or more years of experience, working in public sectors management and in local legislative and executive functions.

In this way, the three sectors of the economy were represented in the panel of respondents and, in general, the three groups formed characterize the main stakeholders of the theme developed in this research, and are displayed in the Table 3. Finally, it is worth mentioning that the names of the specialists will not be disclosed in order to preserve their anonymity, according to the rules determined for conducting research using the Delphi method.

Table 3
Distribution of panelists for e-Delphi research.

Group	Government	Universities	Industries	Guests	Participants
1	X			9	5
2		X		9	5
3			X	7	5

• Elaborated by the authors.

For the panel composition, the experts were invited by prior contact by phone or e-mail and later personal contact, when the following items were exposed:

1. a

   Objective of the study;

2. b

   General description of the e-Delphi method;

3. c

   Form of disclosure of results.

From a methodological point of view, the e-Delphi technique imposes the observation of criteria of validity and reliability. Regarding panelists, it is considered that they must satisfy the following conditions (Hasson & Keeney, 2011):

1. a

   To know personally the region, the community or the institution under study;

2. b

   Perform the functions or develop activities related to the object of the study;

3. c

   Participate actively in the life of the community or the institution;

4. d

   Have participated in previous work on problems related to the subject under study.

In general, a balanced distribution between elements should be sought using universities, research institutes, industries and other sectors of society.

The choice of number of rounds must be based on the finding that the values of consensus change are small between the rounds on certain key issues - the most controversial ones -, also considering the participation emptying when the option is for a high number of rounds (Hasson & Keeney, 2011).

It is worth mentioning that, by Worrell et al. (2012), most works use between ten and thirty specialists, and that a panel with four specialists is appropriate when they have deep knowledge on the subject studied.

8. Preparation, testing and validation of the questionnaires

Careful design of the questions asked requires clarity and precision in the concepts spelled out in each statement, since many of the questions in a research by the Delphi Method are not typical questions of a common research, and in some cases it is necessary to define concepts.

It is enphasized that for this research purposes, “proposition” was conceptualized as a content expressed by a statement, whose truth or falsity can be accepted by a respondent in an intensity of disagreement or null, partial or full agreement. It is further clarified that the test validation of the questionnaire was carried out by three researchers experts in scientific research methodologies.

For the personal declaration of the degrees of consensus or dissent by the panelists, a Likert Scale (Dalmoro and Vieira, 2014) was inserted after each proposition, adapted for the purpose of this work, to an even number of answer options, with the intention of avoid central tendency of positioning, by way of exemption or comfort due to uncertainty on the part of the respondentss. A field for inserting optional comments was also added, as shown in Table 4.

Table 4
Delphi method – Adapted Likert scale.

Strongly disagree	1	2	3	4	5	6	7	8	9	10	Strongly agreement
	()	()	()	()	()	()	()	()	()	()
Respondent comments:

• Souce: Elaborated by the authors.

The questionnaire was made available to the panelists through forms sent by email and the answers were inserted in the same document, and later transferred to an electronic spreadsheet suitable for data tabulation.

The evaluation of the responses presented by the panelists on the Likert Scale, used in this research in the 10-point format (Table 4), was carried out by applying the statistical calculations of the Average (1), the Median (2), the Variance (3), the Sample Standard Variation (4), Concordance Validation Index (5), the Coefficient of Variation (6) and the Interquartile Range (7), whose formulas are shown in the Table 5.

Table 5
Statistical calculations applied to data processing.

Calculations	Formulas		What measures or determines	Expected value range according to the consulted literature
Average	(1)	$\overline{x}=\frac{1}{n}{\displaystyle \sum }_{i-1}^n{x}_i$	The ratio of the sum of all elements in a data set.
Median	(2)	$Med=x_{\frac{n+1}{2}}$	The value that occupies the central position of the series of observations of a variable, in a roll, dividing the set in two equal parts.
Variance	(3)	$S^2=\frac{\sum \left(x-\overline{x}\right)}{n-1}$	Shows the variation around the measure.
Sample standard deviation	(4)	$S=\sqrt{{\displaystyle \sum }_{i=1}^N\frac{{\left(x_1+\overline{x}\right)}^2}{n-1}}$	Shows the distance of each value in the data set in relation to the central mean value. The smaller, the closer the value is to the average.
Concordance Validation Index	(5)	$CVI\left(h\right)=\frac{\sum g}{\sum r}$	The degree of consensus measured about a proposition using n-point adapted Likert scale. Notes: (g = grades assigned) (r = maximum possible score)	≥ 0.7
Coefficient of Variation	(6)	$CV=\frac{\text{S}}{\overline{{\rm X}}}\times 100$	The variation between data set measures on scales. Measures the variation relative to the average.	≤ 30%
Interquartile Range	(7)	$IR=\left(Q_3–Q_1\right)/Q_1$	The degree of a spread of data around the centrality measure. A measure of data dispersion around the population average.	≤ 40%

• Elaborated by the authors.

The expected value ranges for the CVI-h, CV and IR indicators, when applied to consensus and dissent measurements, from the recommendations of the authors, Martuza, (1977); Davis (1992); Lynn, (1986); Rúbio et al. (2003); Wynd et al. (2003); Okoli & Pawlowski, (2004); Polit & Beck, (2006); Alexandre & Coluci, (2011); Miguel (2012) and Beatty (2018), are also shown in Table 5.

Related to statistical treatments, Beatty (2018) warns, “Statistical calculations do not make decisions. The role of all statistics (of any kind) is limited to supporting the analysis that the decision maker needs to make, in order to decide as to what to do in the face of a problem or situation.”

Quantitative research data were processed using Stat Plus LE software (https://www.analystsoft.com/br/products/statplusmacle/), SPSS (https://www.ibm.com/br-pt/analytics/ spss-trials). It is reiterated that the Delphi Method uses scientific techniques that allow the analysis of qualitative and quantitative data. It is a research tool that allows discovering the opinions of specialists, called panelists, on complex issues.

However, opinions may prove to be congruent or divergent, since they are exposed to the panelists' individual perceptions, and may generate a coefficient of agreement or disagreement below expectations.

9. Procedures for obtaining and data analysis

As a contribution through the e-Delphi Method, Linstone & Turoff (2011) and Grisham (2009), it was proposed to a group of specialists rooted in Brazilian Electronic Valley, participation in a research by the Delphi Method, with initial purpose of knowing the structure and effectiveness of the local TH.

A set of tools for collecting information and analyzing data was used to fulfill the specific objectives of the research (Table 6).

Table 6
Methodological tools for collecting and analyzing data and information.

Specific objectives	Collection tools	Analisys tools
Identify and describe the agents of TH existing in the region object of the research.	Field, bibliographic and documentary research.	Document analysis
Identify the current status of the presence of a set of elements considered essential for the recognition of the existence and functionality of the TH model in the region.	Field research	Parametric and non-parametric statistics
		SPSS software
	e-Delphi Surveys and Survey	Stat Plus LE software

• Elaborated by the authors.

To carry out the study at the local level, a group of panelists from institutions representing the three propellers proposed by the TH Model (Etzkowitz & Zhou, 2017) was selected, namely the government, educational institutions and organizations in general present at the researched locus.

It is noteworthy that the option for applying the calculation of three indicators (Concordance Validation Index, Coefficient of Variation and Interquartile Range) as described in the Table 5 to identify the levels of consensus or dissent among specialists, sought to apply greater rigor in the use of the method and greater reliability of the Likert Scale as a measurement instrument used.

Through the Concordance Validation Index, the degree of consensus about a proposition was measured using a 10-point Likert Scale. The average and relative variability between data sets was verified by calculating the Coefficient of Variation, and finally, the degree of spreading of data around the measure of centrality was identified when applying the calculations of the Interquartile Range.

It should be noted that the methodology followed in this research can be used in other contexts, in particular for the evaluation of other TH models or future initiatives for electronic participation, both at the regional and national levels, as well as in other countries.

For the proper functioning of a Triple Helix, several authors (Etzkowitz, 2017; Etzkowitz & Zhou, 2017; Oliveira et al., 2018), indicate to us as fundamental, the institution and the full exercise of organizational leadership, the intermediation roles, the creativity exercise and commitment to innovations.

For the aforementioned authors, research that aims to identify the status of existence and proper functioning of a TH in a region must, among other aspects, identify the presence and performance of the organizational roles of its members.

It is deduced by the exposed that, the credibility of a research for this purpose must not do without on the part of the respondent, his position based on his experience, cooperation and fundamental comments for the understanding of the forms, the intensity and importance of interactions between the agents of the development of TH, integrated by the government agencies, educational institutions and society, for this purpose understood by organizations in general established in the researched region.

In Brazil, the distancing from the institutions constituting the TH model leads to a condition of low systemic maturity, mainly due to the imposition of traditionalist bases of its historical-cultural process, perceived as lethargic in responding to the competitiveness of the world economic scenario, in face of the link to institutionalism statesman who distances the country from modernity (Beck, 2010).

Regarding systemic maturity, Etzkowitz & Zhou (2017), refer to a set of guiding principles, which according to them, allows us to recognize the existence, as well as to identify the solidity, efficacy and effectiveness of a properly constituted and successful TH, as described in the results section.

For Etzkowitz & Zhou (2017) this is a minimum set of guiding principles. However, they assert that others can be inserted from the perception of the need for control, adjustments and better design of activities and the development of the role of each member within the TH model.

10. Propositions and research results

In this chapter, the steps taken to carry out the research using the e-Delphi method and the main results obtained will be presented in detail.

Through a questionnaire (Table 7), fifteen propositions were presented that refer to a set of determining factors developed as essential for the recognition of the existence and consolidation of a Triple Helix in a region.

Table 7
A set of determinant factors for the existence of a Triple Helix.

No.	Synthesis of the guiding determinant factors for the existence of the Triple Helix Model
1	The local TH should demonstrate an intensity of involvement and cooperation between the leaders of each member (University-Industry-Government), as a common objective and decisive factor for its existence and consolidation in a society.
	Less present												I perceive													More present
	1		2		3				4			5				6		7				8			9				10
2	Local educational institutions are present and committed to the development of their region, and in them, a significant number of teachers encourage their graduates to take advantage of and carry forward the technology developed in their laboratories, and these teachers work in parallel, in part or whole, in high-tech companies, as expected from a dynamic and advanced TH model.
	Less present													I perceive													More present
	1		2		3				4			5				6		7				8			9				10
3	Regular meetings are realized between government representatives, educational institutions, and industry, as well as with other community protagonists, to discuss local problems and potential, create bases for the strengthening of relations between TH members, towards to get joint discoveries for the challenges of development and preservation of local hegemony.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
4	In a vibrant TH, there is a continuous interaction between the university, the industries, and the government in the construction of partnerships, agreements, and constant performance in new development strategies and innovation practices resulting from this tripartite cooperation.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
5	The government's role in TH must be a mediator and supporter, acting, acting, among other ways, in creating spaces for the integration of makers, fab labs, public business incubators, and development agencies for innovation projects.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
6	In an active Triple Helix, the open system principle must be considered. For this, civil society needs to be represented as an adjunct to the initiatives of key members University and Government and as an opinion maker in industry research, especially in the field of innovations aimed at new uses and consumption trends.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
7	The existence and actions of a local Triple Helix must be known and accessible to everyone. The premise for the occurrence of collective protagonism must consider the participation of civil society, as a plaintiff or provider of facilitating solutions, while integrated with the processes of sharing open innovations.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
8	The corporate governance of TH must be exercised in a participatory manner. It is recommended to create a council acting as an advisor and mediator in decision-making involving eventual unilateral interests, to preserve reciprocity, involvement, and cooperation between members, seeking to ensure proximity, cordiality, and constructive relationships.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
9	To contribute as a key driver in a knowledge-based economy, and as an important player in social development, educational institutions need to test the theory of their discoveries in organizations in the public and private sectors. This bilateral cooperation should be encouraged and promoted by such sectors, through the provision of structural incentives and financial resources for research and development, especially in the field of innovations.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
10	The identity among the members of the TH must be created through an effective communication process between its three main representatives, University-Industry-Government, who must strip themselves of isolated interests in the political, ideological, and personal fields, to maintain themselves synergistically united around a common objective: development, progress and the existential continuum of cohabited space.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
11	Triple Helix's corporate governance must consider the periodic alternation of the power of direction and representation among its three constituents. For the plurality of perceptions to occur in the generations of effective contributions necessary for the success of the TH model, single, long-term governance must be avoided.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
12	In a TH, the participation of private venture capital based on partnerships depends, among other aspects, on a set of attractive conditions for investments based on the credibility, transmitted and perceived by the investor as to the solidity of the Government, as a promoter and financier of initiatives, of the University as a generator of applicable knowledge and the industry know-how to generate technological innovations and value-added goods.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
13	The performance of a TH in terms of the development of the economy and the progress of a locality does not depend only on its constitution. It requires the presence and sum of efforts promoted by other important peripheral sectors such as business accelerators, technology transfer agencies, export support agencies, cooperatives and unions, private sponsors, and financiers, among others.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
14	To consolidate itself in the region where a Triple Helix is inserted, through its members, it must generate results in terms of enabling solutions, which can create added value for local socio-economic development when compared to other regions where the model does not yet exist.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10
15	A fundamental condition for the consolidation of the University as part of a Triple Helix socially engaging is the existence of resources and the practice of activities that generate technical and scientific knowledge, such as own spaces or structured by partners, where co-working activities, scientific research, symposia, congresses, fairs, and lectures can be carried out.
	Less present										I perceive													More present
	1		2		3				4			5				6		7				8			9				10

• Elaborated by the authors.

In this sense, respondents were asked to, according to their personal perception, record on the Likert Scale displayed in Table 4, the degree of intensity of the presence in the Brazilian Electronics Valley, of each of the factors proposed as determinants of the existence and consolidation of the local TH.

The data recorded by the panelists in the questionnaire were statistically analyzed using the Concordance Validation Index, the Coefficient of Variation and the Interquartile Range, as shown in Table 8.

Table 8
Compilation of the TH's diagnosis results.

COMPILATION OF RESEARCH RESULTS BY THE e-DELPHI METHOD
Panelists			Determining factors of the presence and consolidation of the Triple Helix in the Brazilian Electronics Valley
			1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
GRADES ASSIGNED	UNIVERSITY	1	8	9	7	7	7	8	8	8	8	8	8	8	9	9	9
		2	4	7	2	4	3	6	5	5	6	4	6	3	5	6	7
		3	7	8	4	5	8	6	4	5	4	4	5	5	5	4	8
		4	7	9	7	5	4	5	7	7	7	8	5	5	8	5	7
		5	7	7	5	8	5	6	5	6	7	8	7	5	6	7	9
	INDUSTRY	6	8	9	8	7	5	7	7	8	10	10	8	8	9	10	9
		7	7	7	7	6	10	9	7	8	10	9	10	9	9	8	9
		8	4	5	4	5	7	7	7	6	5	6	6	6	5	4	4
		9	10	10	10	10	9	10	10	10	10	10	8	8	10	7	8
		10	10	10	6	10	10	8	10	10	7	7	10	6	1	5	10
	GOVERNMENT	11	5	3	4	5	6	4	4	5	6	4	5	5	4	5	4
		12	4	7	4	6	6	6	9	4	6	7	6	2	4	10	2
		13	7	8	6	7	5	7	7	7	6	7	8	8	9	8	8
		14	10	10	7	9	7	8	8	8	7	8	7	8	8	7	8
		15	8	10	7	9	7	10	10	9	8	9	7	6	10	8	10
STATISTICAL DATA PROCESSING		CVI	0.7	0.9	0.5	0.5	0.5	0.5	0.7	0.6	0.6	0.7	0.6	0.4	0.5	0.6	0.8
		∑	106	119	88	103	99	107	108	106	107	109	106	92	102	103	112
		VAR	4.4	4.1	4.1	3.8	4.3	3.0	4.2	3.5	3.3	4.1	2.6	4.1	7.3	4.0	5.6
		M	7.1	7.9	5.9	6.9	6.6	7.1	7.2	7.1	7.1	7.3	7.1	6.1	6.8	6.9	7.5
		Ma	7.0	8.0	6.0	7.0	7.0	7.0	7.0	7.0	7.0	8.0	7.0	6.0	8.0	7.0	8.0
		S	2.0	1.9	2.0	1.9	2.0	1.7	2.0	1.8	1.7	1.9	1.6	2.0	2.6	1.9	2.3
		CV	29%	25%	33%	28%	30%	23%	27%	26%	24%	27%	22%	32%	38%	28%	30%
		Q1	6.0	7.0	4.0	5.0	5.0	6.0	6.0	5.5	6.0	6.5	6.0	5.0	5.0	5.0	7.0
		Q3	8.0	9.5	7.0	8.5	7.5	8.0	8.5	8.0	8.0	8.5	8.0	8.0	9.0	8.0	9.0
		IR	33%	36%	75%	70%	50%	33%	42%	45%	33%	31%	33%	60%	80%	60%	29%

• Elaborated by the authors.

According to Table 8, for the degrees of agreement intensity as to the propositions, and confirmation of the determinant factors of the existence of the Triple Helix model in the Brazilian Electronic Valley, the results showed that, in terms of the Concordance Validation Index, the degree of agreement of the panelists positioned themselves below the expected minimum (0.70) for 67% of them.bFor the Coefficient of Variation, 67% of the results were recorded at the maximum expected percentage of 30% and for the Interquartile Interval 53% of the results were above the maximum of 40% expected.

Note that the graphical representation of the compilation of results obtained after the statistical treatment of the quantitative data is recorded in Table 8.

It is observed that the synthesis of the results of the TH diagnosis shown in the Table 8 and in the Figure 4 records that the majority of panelists considered the degree of intensity, presence and consolidation of the TH model in Brazilian Electronic Valley as incipient, when we take as reference (Martuza, 1977; Davis, 1992; Lynn, 1986; Rúbio et al., 2003; Wynd et al., 2003; Okoli & Pawlowski, 2004; Polit & Beck, 2006; Alexandre & Coluci, 2011; Miguel, 2012 and Beatty, 2018).

Figure 4
Compilation of results for the CVI-h Index between the three groups.

The compilation of the results of the three groups of respondents indicated that 67% of the determinants of the existence and consolidation of TH were not confirmed in the surveyed locus. The homogeneity of responses was confirmed among 67% of respondents. However, the percentage of the Interquartile Range indicated high dispersion for 53% of the determining factors listed.

Through the compilation shown in Figure 5, it is possible to verify significant data homogeneity. Hence, the proximity of the panelists' perceptions about the evaluated context rated.

Figure 5
Compilation of results for the Coefficient of Variation between groups.

The percentage of the Interquartile Range according to Figure 6 indicated high dispersion for 47% of the determining factors listed. It is observed that factors 3; 4; 5; 8; 12; 13 and 14 stood out among the others, indicating among other aspects the existence of low involvement and communication failures, elements that can contribute to the lack of synergy among the members of TH.

Figure 6
Compilation of results for the Interquartile Range between groups.

The compilation of the results of the group of universities, as shown in Figure 7, indicated that 67% of the determinants of the existence and consolidation of TH were not recognized by the respondents in the surveyed locus.

Figure 7
Compilation of results for the Concordance Variation Index (Universities).

Through the compilation shown in Figure 8, it is possible to verify a significant homogeneity of the data. It is noted that 73% of the results of the evaluations remained below the expected upper limit. In this sense, it was inferred the existence of proximity of the panelists' perceptions about the evaluated context.

Figure 8
Compilation of results for the Coefficient of Variation (Universities).

The percentage of the Interquartile Range indicated high dispersion for 60% of the determining factors listed. It is observed that factors 1; 3; 5; 6; 9; 10; 12 and 13 stood out among the others, indicating among other aspects the existence of low involvement and communication failures, elements that can contribute to the lack of synergy among the members of TH.

The compilation of the results of the group of universities indicated that 67% of the determinants of the existence and consolidation of TH were not recognized by the respondents in the surveyed locus.

Through the compilation shown in Figure 11, it is possible to verify the predominance of the positioning of the data below the expected maximum limit. Through this result, it was inferred by the existence of alignment of the panelists' perceptions about the evaluated context.

Figure 11
Compilation of results for the Coefficient of Variation Index (Industries).

As can be seen in the Figure 12, among the three groups, industry representatives registered the smallest dispersion (20%) around the result of the coefficient of variation. This reality indicates a condition of greater leveling between their perceptions about the determining factors evaluated.

Figure 12
Compilation of results for the Interquartile Range Index (Industries).

Among government representatives, 67% confirmed through their responses their alignment regarding the incipience of the presence of the factors determining the presence and consolidation of the Triple Helix in the surveyed location.

Through the compilation shown in Figure 14, it is possible to verify that only 33% of the data are positioned above the expected upper limit. Thus, it was observed the occurrence of homogeneity among the panelists about their perceptions about the evaluated context.

Figure 14
Compilatio.n of results for the Coefficient of Variation (Government).

The percentage of the Interquartile Range among government respondents indicated high dispersion for 80% of the determining factors listed. It is observed that factors 1; 3; 4; 5; 8; 9; 10; 11;12; 13 and 14 stood out among the others, indicating among other aspects the existence of low involvement and communication failures, elements that can contribute to the lack of synergy among the members of TH.

11. Summary of results and conclusions

In this article, we present the development and the results of a research carried out to know the modus agendi of a Regional Triple Helix and to measure the intensity of the presence of a set of factors proposed as determinants of its existence and consolidation.

We argue that the credibility of a research for this purpose should not dispense, on the part of the respondent, of its positioning based on its experience, cooperation and fundamental comments for the understanding of the forms, the intensity and the importance of the interactions between the agents of a TH.

We identified that the respondents' opinions may prove to be congruent or divergent, since they are exposed to individual perceptions, and may generate a coefficient of agreement or disagreement below the consulted literature authors expectations.

The results presented may contribute to the investigative processes that aim to confront the modus agendi of a Triple Helix with a set of propositions and premises for its existence and consolidation.

When verifying the results from the applied statistical methods, it was observed a mixture of consensus and dissent among the panelists.

From Figure 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, it can be seen among individuals belonging to the same group, and among the three groups University-Industry-Government, the occurrence of considerable dissension in relation to the proposals presented.

Figure 9
Compilation of results for the Interquartile Range (Universities).

Figure 10
Compilation of results for the Concordance Variation Index (Industries).

Figure 13
Compilation of results for the Concordance Variation Index (Government).

Figure 15
Compilation of results for the Interquartile Range (Government).

It is inferred from the level of the intensity of consensus (CVI-h), the relative variation (CV) and the distancing of the scores attributed by the panelists (IR), the occurrence of perceptions misalignment among them, regarding to the set of determinant factors of the existence and consolidation of the Triple Helix researched. The research results also indicated the existence of a Triple Helix, going a long way towards consolidation.

It is asserted that this status quo corresponds to a strong indicator of incipience relative to the degree of intensity of presence, practice and consolidation of the Triple Helix in the Brazilian Valley of Electronics.

The perception left by the interviewees during field verbal contatcts, indicate that the results were consistent with the reality of the researched region, in relation to the existence of differences between the members of the local Triple Helix, and the need to create and implement an instrument capable of bringing together local organizations and society, in a common objective aligned to the actions to consolidate an intelligent, creative and prosper community.

In this sense, the lack of an instrument capable of promoting the convergence and virtual integration of the members of the three propelling blades, - university - government - society, was chosen as a specific gap for contribution, work and collective achievements.

To fill this identified gap, it is suggested future academic research that seeks to find ways to improve the interaction between University - Industry - Government through instruments that, facilitate consensus while searching shared innovations and solutions.

It is understood that the general objective was achieved, considering that a set of factors proposed as determinants of the existence and consolidation of a TH was presented and evaluated by the members of the TH of the Brazilian Valley of Electronics.

The implications of this research for theory and practice in the academic environment can be considered, mainly, in relation to the unprecedented application of the calculation of three statistical indicators (Concordance Validation Index, Coefficient of Variation and Interquartile Range) for the quantitative data derived from a research by the e-Delphi Method.

Among the advantages observed with the application of the e-Delphi method to measure consensus, the fact that it is possible to carry out the entire process via electronic means of communication emerges. The speed in obtaining the answers, once the systematic monitoring is carried out with the respondents, is also configured among the advantages. On the other hand, it was observed that it is laborious to form a large team of panelists with experience in the researched object.

In sum, it is expected that the determinant factors proposed in this article may assist researchers in future work to verify the existence, solidity, and consolidation of other models of TH present in other regions of the world. It is also considered opportune to instigate the academic community to research aimed at proposing instruments that can contribute to the convergence and virtual integration of the TH's model members.

• How to cite this article: Oliveira, A. H., Marins, F. A. S., Delamaro, M. C. (2021). Concordance measurement applying the Delphi method: proposal and evaluation of a set determinants factor of the Triple Helix existence. Production, 31, e20200110. DOI: https://doi.org/10.1590/0103-6513.20200110

References

• Alexandre, N. M. C., & Coluci, M. Z. O. (2011). Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciência & Saúde Coletiva, 16, 3061-3068. https://www.scielo.br/j/csc/a/5vBh8PmW5g4Nqxz3r999vrn/abstract/?format=html⟨=pt
  » https://www.scielo.br/j/csc/a/5vBh8PmW5g4Nqxz3r999vrn/abstract/?format=html⟨=pt
• Ameyaw, E. E., Hu, Y., Shan, M., Chan, A. P. C., & Le, Y. (2016). Application of Delphi method in construction engineering and management research: a quantitative perspective. Journal of Civil Engineering and Management, 22(8), 991-1000. http://dx.doi.org/10.3846/13923730.2014.945953
  » http://dx.doi.org/10.3846/13923730.2014.945953
• Beatty, W. (2018). Four facts about all Statistics. In W. Beatty (Ed.), Decision support using nonparametric statistics (Springer Briefs in Statistics). Cham: Springer.http://dx.doi.org/10.1007/978-3-319-68264-8_2.
• Beck, U. (2010). Individualization, institutionalization and standardization of living conditions and biographical models. In U. Beck, Risk society: towards another modernity São Paulo: Editora 34. Translated by Sebastião Nascimento.
• Chung, C. J., & Park, H. W. (2014). Mapping Triple Helix innovation in developing and transitional economies: webometrics, scientometrics, and informetrics. Scientometrics, 99(1), 1-4. http://dx.doi.org/10.1007/s11192-013-1105-6
  » http://dx.doi.org/10.1007/s11192-013-1105-6
• Cortez, R. M., & Johnston, W. J. (2017). The future of B2B marketing theory: A historical and prospective analysis. Industrial Marketing Management, 66, 90-102. http://dx.doi.org/10.1016/j.indmarman.2017.07.017
  » http://dx.doi.org/10.1016/j.indmarman.2017.07.017
• Creswell, J. W. (2014). A concise introduction to mixed methods research Los Angeles: Sage Publications.
• Dalmoro, M., & Vieira, K. M. 2014. Dilemmas in the construction of Likert-type scales: do the number of items and the disposition influence the results? Revista Gestão Organizacional, 6(3), 161-174. Available in: https://scholar.google.com.br/scholar?q=escala+de+likert+Dalmoro&hl=pt-BR&as_sdt=0,5
  » https://scholar.google.com.br/scholar?q=escala+de+likert+Dalmoro&hl=pt-BR&as_sdt=0,5
• Davis, L. L. (1992). Instrument review: Getting the most from your panel of experts. Applied Nursing Research, 5(4), 194-197. http://dx.doi.org/10.1016/S0897-1897(05)80008-4
  » http://dx.doi.org/10.1016/S0897-1897(05)80008-4
• De Carli, P. C., Delamaro, M. C., & Salomon, V. A. P. (2010). Identification and prioritization of critical success factors in a digital factory implementation. Production, 20(4), 549-564.
• Eberhart, M. E., & Pascuci, L. 2014. The decision-making process and its implications for university, business and government cooperation: a case study. Revista GUAL, 7(2), 221-242. Available at: https://periodicos.ufsc.br/index.php/gual/article/view/31428
  » https://periodicos.ufsc.br/index.php/gual/article/view/31428
• Etzkowitz, H. (2013). Triple Helix: University-industry-government: innovation in motion Porto Alegre: EDIPUCRS, 2013.
• Etzkowitz, H. (2017). Innovation Lodestar: The entrepreneurial university in a stellar knowledge firmament. Technological Forecasting and Social Change, 123, 122-129. Available in: https://scholar.google.com.br/scholar?hl=pt-BR&as_sdt=0%2C5&q=The+entrepreneurial+university+in+a+stellar+knowledge+firmament.&btnG
  » https://scholar.google.com.br/scholar?hl=pt-BR&as_sdt=0%2C5&q=The+entrepreneurial+university+in+a+stellar+knowledge+firmament.&btnG
• Etzkowitz, H., & Zhou, C. (2017). Triple Helix: Innovation and entrepreneurship university-industry-government. Advanced Studies, 31(90), 23-48. Available in: http://www.scielo.br/pdf/ea/v31n90/0103-4014-ea-31-90-0023.pdf
  » http://www.scielo.br/pdf/ea/v31n90/0103-4014-ea-31-90-0023.pdf
• Giannarou, L. & Zervas, E. (2014). Using Delphi technique to build consensus in practice. International Journal of Business Science & Applied Management, 9(2), 65-82.
• Grisham, T. (2009). The Delphi technique: a method for testing complex and multifaceted topics. International Journal of Managing Projects in Business, 2(1), 112-130. Available in: https://www.emerald.com/insight/content/doi/10.1108/17538370910930545/full/html
  » https://www.emerald.com/insight/content/doi/10.1108/17538370910930545/full/html
• Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695-1704. Available in: https://www.sciencedirect.com/science/article/pii/S0040162511000801?via%3Dihub
  » https://www.sciencedirect.com/science/article/pii/S0040162511000801?via%3Dihub
• Holland, S., & Oliveira, T. C. (2014). The Rise, decline, and regeneration of industrial districts: Enhanced HRM, SMEs, and location. In Machado, C. & Melo, P. Effective human resources management in small and medium enterprises: Global perspectives (pp. 238-279). Hershey, PA: IGI Global. Available in: https://www.igi-global.com/chapter/the-rise-decline-and-regeneration-of-industrial-districts/93211
  » https://www.igi-global.com/chapter/the-rise-decline-and-regeneration-of-industrial-districts/93211
• Kapetaniou, C., & Lee, S. H. (2017). A framework for assessing the performance of universities: the case of Cyprus. Technological Forecasting and Social Change, 123(1), 169-180. http://dx.doi.org/10.1016/j.techfore.2016.03.015
  » http://dx.doi.org/10.1016/j.techfore.2016.03.015
• Kim, M., Jang, Y. C., & Lee, S. (2013). Application of Delphi-AHP methods to select the priorities of WEEE for recycling in a waste management decision-making tool. Journal of Environmental Management, 128, 941-948. Available in: http://www.irantahgig.ir/wp-content/uploads/40037.pdf
  » http://www.irantahgig.ir/wp-content/uploads/40037.pdf
• Kobza, N., & Mutlucan, C. (2016). Entrepreneurship leading a change in Europe: a perspective of young professionals. IFAC-PapersOnLine, 49(29), 289-293. Available in: https://www.sciencedirect.com/science/article/pii/S2405896316325198
  » https://www.sciencedirect.com/science/article/pii/S2405896316325198
• Landeta, J., Barrutia, J., & Lertxundi, A. (2011). Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts. Technological Forecasting and Social Change, 78(9), 1629-1641. Available in: http://www.sciencedirect.com/science/article/pii/S0040162511000618
  » http://www.sciencedirect.com/science/article/pii/S0040162511000618
• Lawthon, S. H., & Leydesdorff, L. (2014). The Triple Helix in the context of global change: dynamics and challenges. Prometheus, 32(4), 321-336. Available in: https://www.tandfonline.com/doi/abs/10.1080/08109028.2014.972135
  » https://www.tandfonline.com/doi/abs/10.1080/08109028.2014.972135
• Lee, Y. H., & Kim, Y. (2016). Analyzing interaction in R&D networks using the Triple Helix method: Evidence from industrial R&D programs in Korean government. Technological Forecasting and Social Change, 110, 93-105. Available in: https://paperdownload.me/wp-content/uploads/2017/10/4303-analyzing-interaction-rd-networks-triple-helix-method-industrial-rd-programs-korean-government.pdf
  » https://paperdownload.me/wp-content/uploads/2017/10/4303-analyzing-interaction-rd-networks-triple-helix-method-industrial-rd-programs-korean-government.pdf
• Leydesdorff, L. (2013). Triple Helix of university-industry-government relations. In E.G. Carayaniss (Ed.), Encyclopedia of creativity, invention, innovation and entrepreneurship (pp. 1844-1851). New York, NY: Springer. http://dx.doi.org/10.1007/978-1-4614-3858-8_452
  » http://dx.doi.org/10.1007/978-1-4614-3858-8_452
• Linstone, H. A., & Turoff, M. (2011). Delphi: A brief look backward and forward. Technological Forecasting and Social Change, 78(9), 1712-1719. Available in: https://www.sciencedirect.com/science/article/pii/S0040162510002155
  » https://www.sciencedirect.com/science/article/pii/S0040162510002155
• Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382-385. http://dx.doi.org/10.1097/00006199-198611000-00017 PMid:3640358.
  » http://dx.doi.org/10.1097/00006199-198611000-00017
• Manzini, E. (2014). Making things happen: Social innovation and design. Design Issues, 30(1), 57-66. http://dx.doi.org/10.1162/DESI_a_00248
  » http://dx.doi.org/10.1162/DESI_a_00248
• Martuza, V. R. (1977). Applying norm-referenced and criterion-referenced measurement in education Boston: Allyn & Bacon.
• Miguel, P. A. C. (2012). Research methodology in production engineering and operations management (2nd ed.). Rio de Janeiro: Elsevier Editora.
• Nogueira, V. D. C., & Fuscaldi, K. (2018). Painel de especialistas e Delphi: métodos complementares na elaboração de estudos de futuro Brasília, DF: Embrapa. Available in: https://www.infoteca.cnptia.embrapa.br/infoteca/handle/doc/1091361
  » https://www.infoteca.cnptia.embrapa.br/infoteca/handle/doc/1091361
• Oliveira, A. H. (2012). Inserções no Modelo da Tríplice hélice: um estudo de caso na incubadora do Inatel. In XII Encontro Latino Americano de Pós-Graduação São José dos Campos, São Paulo: Universidade do Vale do Paraíba. http://www.inicepg.univap.br/cd/INIC_2012/anais/trabalhos_sociaisaplicadas.html
  » http://www.inicepg.univap.br/cd/INIC_2012/anais/trabalhos_sociaisaplicadas.html
• Oliveira, A. H., Marins, F. A. S., Delamaro, M. C. (2018). Business engineering incubators: a cooperation case for cluster’s development. Production, 28, 1-16. Available in: http://dx.doi.org/10.1590/0103-6513.20180001.
  » https://doi.org/10.1590/0103-6513.20180001
• Okoli, C., & Pawlowski, S. D. 2004. The delphi method as a research tool: an example, design considerations and applications. Information & Management, 42(1), 15-29. Available in: https://spectrum.library.concordia.ca/976864/1/OkoliPawlowski2004DelphiPostprint.pdf
  » https://spectrum.library.concordia.ca/976864/1/OkoliPawlowski2004DelphiPostprint.pdf
• Pellin, D. R., & Engelmann, W. A. (2018). Economic analysis of the law of individual microentrepreneurs beyond the law: politics. Economic Analysis of Law Review, 9(2), 177-193.
• Polit, D. F., & Beck, C. T. (2006). The content validity index: are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497. Available in: https://scholar.google.com.br/scholar?hl=pt-BR&as_sdt=0%2C5&q=The+content+validity+index%3A+are+you+sure+you+know+what%27s+being+reported%3F+Critique+and+recommendations.+&btnG=
  » https://scholar.google.com.br/scholar?hl=pt-BR&as_sdt=0%2C5&q=The+content+validity+index%3A+are+you+sure+you+know+what%27s+being+reported%3F+Critique+and+recommendations.+&btnG=
• Rúbio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work. Social Work Research, 27(2), 94-104. http://dx.doi.org/10.1093/swr/27.2.94
  » http://dx.doi.org/10.1093/swr/27.2.94
• Strasser, A. (2017). Delphi method variants in information systems research: Taxonomy development and application. The Electronic Journal of Business Research Methods, 15(2), 120-133. Available in: http://www.ejbrm.com/issue/download.html?idArticle=467
  » http://www.ejbrm.com/issue/download.html?idArticle=467
• Sum, N. L., & Jessop, B. (2013). Competitiveness, the knowledge-based economy and higher education. Journal of the Knowledge Economy, 4(1), 24-44. http://dx.doi.org/10.1007/s13132-012-0121-8
  » http://dx.doi.org/10.1007/s13132-012-0121-8
• Verlinde, S., & Macharis, C. (2016). Innovation in urban freight transport: The triple Helix model. Transportation Research Procedia, 14, 1250-1259. Available in: https://www.sciencedirect.com/science/article/pii/S2352146516301983
  » https://www.sciencedirect.com/science/article/pii/S2352146516301983
• Vlaisavljevic, V., Medina, C. C., & Van Looy, B. (2020). The role of policies and the contribution of cluster agency in the development of biotech open innovation ecosystem. Technological Forecasting and Social Change, 155, 119987. http://dx.doi.org/10.1016/j.techfore.2020.119987
  » http://dx.doi.org/10.1016/j.techfore.2020.119987
• Woodside, A. G. (2010). Case study research: Theory, methods and practice United Kingdom: Emerald Group Publishing.
• Worrell, J. L., Di Gangi, P. M., & Bush, A. A. (2013). Exploring the use of the Delphi method in accounting information systems research. International Journal of Accounting Information Systems, 14(3), 193-208. http://dx.doi.org/10.1016/j.accinf.2012.03.003
  » http://dx.doi.org/10.1016/j.accinf.2012.03.003
• Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25(5), 508-518. http://dx.doi.org/10.1177/0193945903252998 PMid:12955968.
  » http://dx.doi.org/10.1177/0193945903252998
• Zhai, B., & Ng, M. K. (2013). Urban regeneration and social capital in China: A case study of the Drum Tower Muslim District in Xi’an. Cities, 35, 14-25. https://doi.org/10.1016/j.cities.2013.05.003.
  » https://doi.org/10.1016/j.cities.2013.05.003
• Yin, R. K. (2017). Case study research and applications: Design and methods. Thousand Oaks, CA: Sage Publications.

Publication Dates

• Publication in this collection
  20 Aug 2021
• Date of issue
  2021

History

• Received
  03 Nov 2020
• Accepted
  28 June 2021