Application of a Multi-method to Identify a Research Problem

Aplicação de um Método Múltiplo para Identificar um Problema de Pesquisa

ABSTRACT

Context:

there is a certain difficulty for students in understanding what steps need to be followed to guarantee that a chosen research problem is academically valid. There are also difficulties in executing, training, and passing on the methodological procedure.

Objective:

the present study aims to detail the operationalization of a method of identifying research problems, allowing the students to prove the unique and singular character of their research.

Methods:

it is a qualitative, interpretive, and applied research, which uses the action research method with the participatory intervention by one of the authors of this paper, in the definition and guidance on how to use a multi-method to identify a research problem.

Results:

the main contribution of this research is to describe, with examples for each stage, a step-by-step procedure explaining how to perform a multi-method to identify a research problem that helps students operationalize it correctly and systematically.

Conclusion:

following the guidelines proposed in this paper, students are able to bring up real problems from both academic and managerial perspectives, as the method will be executed properly. It also allows teachers to better train their students about how to properly use the multi-method detailed in this paper.

Keywords:
research problem; theoretical-practical gaps; multi-method; methodological execution

RESUMO

Contexto:

há certa dificuldade dos alunos em compreender, e dos professores em ensinar, quais etapas precisam ser seguidas para garantir que um problema de pesquisa escolhido seja academicamente válido.

Objetivo:

o presente estudo visa a detalhar a operacionalização de um método de identificação de problemas de pesquisa, permitindo aos alunos comprovar o caráter único e singular de suas pesquisas.

Métodos:

trata-se de uma pesquisa qualitativa, interpretativa e aplicada, que utiliza o método pesquisa-ação, com a intervenção participativa de um dos autores deste artigo, na definição e orientação de como utilizar um método múltiplo para identificar um problema de pesquisa.

Resultados:

a principal contribuição desta pesquisa é descrever, com exemplos para cada etapa, um procedimento passo a passo explicando como executar um método múltiplo para identificar um problema de pesquisa que ajude os alunos a operacionalizá-lo de forma correta e sistematizada.

Conclusão:

seguindo as orientações propostas neste artigo, os alunos são capazes de trazer problemas reais, tanto do ponto de vista acadêmico quanto do gerencial, na medida em que o método será executado corretamente. Também permite que os professores treinem melhor seus alunos sobre como usar adequadamente o método múltiplo detalhado neste artigo.

Palavras-chave:
problema de pesquisa; lacunas teórico-práticas; método múltiplo; execução metodológica

INTRODUCTION

The correct use and practical application of methods or methodologies in academic research, and not only its theoretical understanding, is of significant relevance for the correct performance and reliability of research. Even more relevant is when a methodological procedure is used to identify the research problem itself, giving it support and reliability, and attesting that a supposed problem is in fact an academic problem. The methods need a correct definition and understanding so that they can be applied empirically (Zappellini & Feuerschütte, 2015).

The methodological aspects of research enable researchers to understand the correct meaning presented by the facts, going beyond its simple observation and registration. The methodology of a scientific research is configured as a way of constructing valid and reliable knowledge, assuming a role as important as knowledge per se (Fernandes, 2017; Zappellini & Feuerschütte, 2015).

Braga (2005) and Moraes, Vale, and Araújo (2013) corroborate this understanding by stating that it is not any ‘not knowing’ that can be treated as research, and there is a clear distinction between research and study - to do research it may be necessary to do a study first. The lack of information, ignorance, or uncertainties about a certain topic or subject is not, in itself, a research problem.

Discussions on how to identify a research problem are constant. However, its operationalization in practice still raises doubts among students and researchers, justifying the object of this research. After all, every researcher seeks to produce interesting, influential, and impactful content, so that the research question is one of the most critical points of a research (Alvesson & Sandberg, 2011).

This paper is relevant in that it details the execution stages of the theoretical multi-method created by Farias, Marchisotti, Maggessi, and Miranda (2019) aiming to identify a research problem. It opens space for the discussion of how to execute this method properly, in a practical and detailed manner, mitigating risks of failures in its operationalization that bring negative impacts on the quality of academic studies. Thus, this paper seeks to answer the following research question: How to identify a research problem, by detailing the qualitative and quantitative steps of a multi-method created for this purpose? There is a lack of studies that address the standardization of methodological procedures, in the various areas of knowledge; there are few works involving multi-methods and that include examples of their use (Oliveira, Magalhães, & MisueMatsuda, 2018; Oliveira & Zanotti, 2018; Paranhos, Figueiredo, Rocha, Silva, & Freitas, 2016).

There is a certain difficulty in understanding what steps must be followed to make sure that a particular research problem chosen is academically valid. At the same time, there are also difficulties in performing, training, and passing on these methodological procedures to new students and researchers (Cooper & Schindler, 2016; Hällgren, 2012; Lana, Partyka, Alberton, & Marcon, 2018). Therefore, this article has as its target audience undergraduate and graduate students.

RESEARCH METHODOLOGY

As for the purpose, this is an applied research, as it aims to generate knowledge that will be used in a practical way, in the researchers’ daily lives, complementing and deepening their knowledge. Regarding the approach, it is classified as a qualitative research, because the suggestion of procedures to be performed, and especially the way to do it, comes from the information and suggestions of the researchers who used it. It has a subjective interpretation nature, not using statistical methods or analysis of quantifiable variables (Gray, 2016; Tani, 1992).

Finally, in terms of objective, the work is characterized as interpretive, since it seeks to explore the authors’ personal experiences with researchers, considering their perspectives and views on how to perform a multi-method of research to identify a research problem proposed by Farias et al. (2019). It is important to mention that the original paper from Farias et al. (2019) is based on theoretical assumptions for the multi-method, which now, in the present paper, focuses on its execution into practice, explaining step by step how to do it in any research, by any researcher.

As for the procedures and research strategy, the work uses action research, according to Figure 1, since, to reach the appropriate form of operationalization of the multi-method, there was participatory intervention of the researchers with the reality, either in the operationalization of their own studies or in the guidance of other students and researchers (Creswell, 2016; Gray, 2016).

Figure 1
Elaboration process flow for the procedure for executing the proposed method.

Thus, the procedures to be described in the next chapter were based on personal experiences of executing the method in their work, as well as on the participatory interaction of the creator of the method by Farias et al. (2019) over 22 years of guidance in identifying problems in researching monographs, dissertations, theses, and academic papers, with more than 300 students, of which the most recent can be cited: Marchisotti (2021), Toledo (2020), and Lordelo (2019). From the mistakes and successes, it was possible to identify the most appropriate and best understood way of executing the mentioned method in a simple, objective, and assertive way.

APPLICATION OF THE MULTI-METHOD

According to the multi-method proposed by Farias et al. (2019), once the topic to be researched is defined, there is the need to perform three important stages, in order to identify gaps to be subsequently adopted as research problems: (1) theoretical gaps - theoretical research; (2) practical gaps - practical research; and (3) theoretical-practical gaps - converging points between theoretical and practical gaps, according to Figure 2. To do so, a step-by-step approach is proposed on how to execute such method, detailing the phases associated with the execution of each of these three stages.

Figure 2
Methodological path.

For Stage 1 - theoretical research - it is necessary to detail the procedure adopted in three steps: (a) in bibliometrics to obtain the most relevant papers for the topic researched by the researcher; (b) in lexical and content analysis of the selected papers; and (c) in the identification of potential theoretical gaps. For Stage 2 - practical research - it is also necessary to detail de procedure adopted in three steps: (a) in execution of the critical incident technique - CIT (Flanagan, 1973); (b) in the content analysis - or in the execution of grounded theory, since both can be used; and (c) in the identification of potential practical gaps. Finally, in Stage 3, it is necessary to compare the theoretical with practical gaps, identified in the Stages 1 and 2, in search convergent gaps between academy and the professional practice, including the needs of public sector, civil society, organizations, and social movements.

In order to make clearer all the procedures that have been suggested in this paper, examples taken from a doctoral dissertation by Marchisotti (2021), who researched on the governance system and adopted the multi-method approach that has been described in detail in this paper, have been included.

Marchisotti (2021) identified 30 theoretical gaps, 24 practical gaps, and eight theoretical-practical gaps, choosing one practical gap as the problem research - stakeholder’s perception that the governance system does not add value to the company’s results/performance. In this paper, we include an extract of ten theoretical gaps, ten practical gaps, and one theoretical-practical gap to be used as an example at the Stage 1.

Stage 1 - Theoretical research

Based on the conceptual model described in Figure 2, this item details the methodological procedures of theoretical research, more specifically how to carry out bibliometrics, lexical, and content analysis.

Thus, Stage 1 consists of three phases - exploration, filter, and consolidation, according to Figure 3. This step of the studied method is aligned with Alvesson and Sandberg (2011; 2013), whose method of defining the research problem based on theoretical gaps is the most widely used in academic studies. This ensures a mass of relevant papers on the researched topic, both for inclusion in the theoretical framework and for the identification of theoretical gaps.

Figure 3
Methodological details of Stage 1 of the studied multi-method.

Then, each of these three phases will have its execution described, making it possible for any interested party to reproduce it.

Phase 1 - Exploration

This phase seeks to identify, using the keyword tree technique (Lacerda, Ensslin, & Ensslin, 2012), which are the most relevant words that should be included in the Boolean rule, according to the research field chosen by the researcher. This Boolean rule will be applied in academic search databases - preferably Scopus and Web of Science, which are widely consulted by researchers and more used for research evaluations (Martín-Martín, Orduna-Malea, Thelwall, & López-Cózar, 2018) - to identify the most relevant documents on the researched topic. The tree of keywords is elaborated from a transversal scan of the literature in search of the thematic distribution of the research theme, in a broad way. For this purpose, it is necessary to perform three activities: (a) Construction of the keyword tree, using Microsoft Word (Morgado, 2017) or similar. With the central theme of the study chosen, the researcher must break it down into thematic axes, which have the role of delimiting bibliographic research, according to Figure 4.

Figure 4
Example of a keyword tree involving the theme: Governance in public sector.

(b) Definition of the Boolean formula to be used in academic search databases, using Microsoft Word (Morgado, 2017) or similar, based on the keyword tree created in activity (a). The terms or expressions in quotation marks that are in the vertical position expand the scope of the search (OR), and the ones in the horizontal are more restrictive (AND), according to Figure 5.

Figure 5
Example of a Boolean formula from the keyword tree.

(c) Exploratory search for papers through the application of the Boolean formula defined in activity (b), in the academic search databases - preferably, but not limited to the Scopus and Web of Science databases, due to their quality and scope -, with subsequent download and importation into EndNote Web software (Hupe, 2019) or similar, building an initial library of papers. Filter the papers published in the last five years and from peer-reviewed journals.

At the end of this step, the researcher will have downloaded a wide range of papers to computer, importing them into EndNote Web (Hupe, 2019), in order to facilitate the execution of the steps of Phase 2 - filter.

Phase 2 - Filter

In this phase, the aim is to eliminate duplicate records, as well as applying filters based on score distributed from four criteria - number of paper citations, productivity of the first author paper, academic journal evaluation, and thematic alignment - in order to obtain a more robust, relevant, and consistent bibliographic information, aligned with the chosen research topic. It is expected that in this phase, a maximum of 90% of the papers obtained in Phase 1 will be eliminated. For this purpose, it is necessary to carry out three activities:

1. Eliminate duplicate papers using the resources of the EndNote Web software (Hupe, 2019).

2. Perform a cross-sectional reading of the title and summary of the papers, in order to eliminate those that are not directly related to the research topic.

3. Export a report of the paper from EndNote Web (Hupe, 2019) to Microsoft Excel (David, 2017), creating a new formatting style called ‘Excel Style,’ so that the following information can be exported: authors, year, title, academic journal, summary, and keywords. Then, the multicriteria CPP method (probabilistic composition of preference method) is applied, according to Table 1 (Treinta, Farias, Sant’Anna, & Rabelo, 2014), giving scores for each of the four proposed criteria. It is necessary to use scales, that is, after multiplying the indicator column by weight, the result is converted to an odd scale from one to nine - one, three, five, seven, nine.

Table 1
Criteria, objectives, and indicators used to filter papers.

Criteria	Objective	Indexes	Contribution
Number of articles’ citations	Consider how significant the article has been considered, from an academic point of view, since citations by other authors show that it is a relevant article.	Number of citations received for the article.	0.23
Productivity of the 1st author´s article	Identify authors who have a consistent academic production, with great productivity and relevance.	Value of the h-index of the articles’ first author.	0.13
Academic journal evaluation	Identify relevant magazines, which publish in quantity and quality, in order to become a reference.	Value of the SJR (SCImago Journal Rank) and SNIP (Source Normalized Impact Per Paper) indexes of the academic journal in which the article was published.	0.22
Thematic alignment	Check the article’s importance, considering its potential contributions to the research theme, according to the researcher’s point of view.	Score given by the researcher, according to the alignment of the article with the research theme.	0.42

• Note. Adapted from Marchisotti (2021). Instead of using the Scopus indicators - SJR and SNP -, you can use the Web of Science indicator - JCR as the journal’s evaluation source, or even the average of the combination of both.

To identify the h-index of the first author and the number of citations of the paper, the researcher should look for the author’s name in Google Scholar (Silva & Grácio, 2017), by clicking on the author’s name, in one of the papers written by him and identified by the search engine. Be careful with homonymous authors. To identify the SJR and SNIP indexes, simply access the Scopus website (Scopus, 2018), informing the name of the academic journal or its ISSN, within the sources tab. Regarding the last criterion - thematic alignment, as this is an analysis that varies from researcher to researcher, it is suggested to use the Table 2 as a reference for the scores, after reading the title and summary of the papers.

Table 2
Criteria and scores for papers’ alignment with research theme.

Criteria	Score
Article not aligned with the research theme	1
Article little aligned with the research theme	3
Article moderately aligned with the research theme	5
Article very aligned with the research theme	7
Article extremely aligned with the research theme	9

• Note. Source: Quelhas (2017).

In Appendix A, Table A1, there is an example of the calculation to be done in paper obtained in activity (b) of this phase, including the final score received by the paper. The journal impact factor index, from the Web of Science (Clarivate, 2020), or the h5-index from Google Scholar (Silva & Grácio, 2017), can be used, to the detriment or in conjunction (average) with the SJR and SNIP indexes detailed here (Martín-Martín et al., 2018).

It is worth mentioning that the researcher should, in addition, search for other bibliographic references that may not have been identified by bibliometrics, using search engines such as Google Scholar and other databases not used in Phase 1. Works that prove to be relevant should be included, such as seminal papers, books, and other sources of knowledge, provided that all their content is available digitally. All papers found in this complimentary research can be incorporated in the research, with a proper justification. It is also worth revising the Boolean rule, as the number of complementary papers found is too large or too small. Eventually, the Boolean rule has some error that needs to be corrected, so that is in fact possible to identify the papers associated with the area and the researched topic.

At the end of this Phase 2, the researcher will have filtered and considerably reduced the number of papers, in order to make it possible to consolidate those of real importance for the researched topic and which will have their content effectively analyzed. Furthermore, it will be possible to obtain a descending order ranking of the papers and their respective final scores.

Phase 3 - Consolidation

In this phase, the aim is to consolidate the papers that are more strongly associated with the research theme and that will have their content analyzed in depth. For this purpose, it is necessary to carry out three activities:

(a) Select the papers that scored seven or nine in the final evaluation of the multicriteria method CPP - Phase 2, with subsequent identification of the first quartile of the sample. To do so, the following procedures must be followed: (1) identify the third quartile of sample N - number of papers in Phase 2, that is, Q³ = 0.75 * (N + 1), and (2) identify the first quartile of sample N, that is, Q¹ = N-Q³, according to Figure 6. Q¹ = N-Q³, according to Figure 6.

Figure 6
Example of calculating the criteria for a paper, according to the CPP method.

(b) The papers selected in activity (a), referring to the first quartile of the sample, should be imported into the NVIVO 11 software (Nvivo, 2017), with subsequent lexical and content analysis, keeping only nouns, verbs, and adjectives. It is expected that this phase will map around 30 lexicons most frequently presented in the analyzed papers and it is suggested that they be presented in two different ways, according to Figure 7. These different ways of presentation help the researcher in the analysis of the results.

Figure 7
Example of three different ways of presenting the lexical analysis of the papers, with the support of the NVIVO 11 software.

(c) Perform the content analysis - codes and categories - of the most frequently identified lexicons in activity (b), still with the support of the NVIVO 11 software (Nvivo, 2017), in order to be able to understand the meaning of these lexicons. It is assumed that it is reasonable to find in a study a minimum of 15 and a maximum of 30 categories, and a minimum of 50 and a maximum of 300 codes.

To perform the content analysis, the first step is to do the data coding. According to Hatch (2002) and Saldaña (2015), codification is the synthesis of a certain analyzed text, to identify patterns. These codes have different characteristics and relationships between them - similarities and differences, frequency of occurrence, the sequence of occurrence, correspondence, and cause and effect -, which, once combined, will be treated as a category. The codes can be represented by a word or phrase, expressing the essence of the data analyzed.

The code represents the data as they are, but in synthesized form. Altogether, there is a minimum of three coding cycles. The category, in turn, is created from the analysis of the convergence of certain characteristics of the various mapped codes, as represented in Figure 8.

Figure 8
Path from raw data collected to categorization, through coding.

In the first coding cycle, the aim was only to create the codes from the words identified by the lexical analysis, unifying those words that had similarity of meaning, such as ‘organizations,’ ‘organization,’ and ‘organizational.’ Then, in the second cycle, the contexts in which the words were used are analyzed, in order to create subcodes that explain the meaning of those words, in order to clearly identify the meaning that the word has in the context of the papers. Emphasis is given to the meaning of the word within the objective of the paper, that is, how the word was involved in the research problem of the paper and all the developments and discussions with which it was associated. Thus, a mere translation of the meaning of the word should be avoided, which would not bring up the subjects covered by the papers and involving the words. In the third cycle, the repeated codes or those with similar content must be grouped together, and the others revised, which has already allowed a visualization of possible categories and more adherent categories, according to Table 3.

Table 3
Example of content analysis based on lexical analysis.

Gap number	Most frequent words	Category inferred	Meaning
1	Public	Dominant public service	Public policies need to be transformed into public services, which are implemented by public companies through projects.
2	Service/Services		The new approaches to the public employees that should be oriented to service provision and meeting the needs and wishes of end users.
3	Governance	Good governance	New demand in the way in which public management should behave, generating good public governance through appropriate control mechanisms.
4	Management	Performance management	New public management focused on managing the company and employees performance.
5	Performance
6	Government	NA	There were no expressions that stood out, since the word government was used in various ways and in different contexts.
7	Project / Projects	Project management	The correct Project management in the public sector increases the implementation rate of the company’s strategy, with potential performance improvement.
8	Political	Political activity	Decision-making in public companies must be independent of political interference when it is harmful. The focus should be on improving performance and service delivery and not serving political interests.
9	Sector	Public partnerships	The public sector needs to build bridges with other sectors, acting collaboratively and complementarily, in order to improve its performance and the quality of its service.
10	Private		There is a need to make public-private partnerships, combined resources, cultures, values and competence, since public services are more modern and have more complex problems.

• Note. Source: Marchisotti (2021).

At the end of Step 1, the researcher will have identified a select group of papers that are relevant and aligned with the research topic, which will be part of the backbone of his theoretical framework. In addition, a series of gaps or theoretical attention points were identified, based on the findings of the analysis of the papers, which can be used as research problems.

Stage 2 - Practical research

At this stage, the objective is to obtain practical gaps, that is, gaps originated from the market. Thus, based on the conceptual model proposed by Farias et al. (2019), it is detailed how to perform CIT and content analysis as part of the second stage - practical research - of implementing the studied multi-method, according to Figure 2.

Step 2 consists of three phases - critical scenario, interviews, and consolidation, according to Figure 9. This ensures a set of relevant data on the researched topic is guaranteed, based on what was exposed by the interviewed specialists, so that it is possible to find the gaps on the researched topic, considering a practical view of the market. This stage of the studied method is aligned with the method of defining the research problem based on the ‘practical need or example,’ as approached by Hällgren (2012), who identifies a research problem from an empirical need, without analyzing the theoretical assumptions on which it is based.

Figure 9
Methodological details of Stage 2 of the studied multi-method.

It is important to highlight that for the use of CIT to be successful, the interviewee needs to have past experience in the research topic, since the critical incident aims only to awaken or explore the opinion of the interviewee about the researched topic, which would be poor in case of inexperienced interviewees. Thus, the concept of oral history is applied, that is, it seeks to capture and analyze the social context of the interviewees in the interviews, through the experiences lived by them on the theme to be researched, within a context presented - critical incident (Padilha & Borenstein, 2005).

Subsequently, each of the three phases for the identification of practical gaps will be described in detail.

Phase 1 - Critical scenario

This phase seeks to collect information about specialists in the researched area, through the definition of a critical scenario on the researched topic and a set of questions, identifying the problems and attention points in the daily practice of the market. It is important to emphasize that the text of the critical scenario should be carefully prepared, considering the theme to be researched, in order to encourage the respondents to express their point of view as much as possible. As a reference, this critical scenario and the questions should occupy a maximum of one page. For this purpose, it is necessary to carry out four activities:

(a) Choosing a more appropriate definition on the researched topic - As the first paragraph of the critical scenario, a definition of the researched theme is inserted, so the interviewee can contextualize which perspective of the topic is being addressed.

(b) Elaboration of the critical scenario - From the second paragraph on, a critical or doubtful scenario involving the researched topic should be described, so the interviewee feels challenged to explain the reasons for the described failure or success. At the end of the critical scenario, at least four questions should be inserted in order to encourage the interviewee to report a personal explanation regarding the researched topic. The interviewee should not be informed of the researcher’s objective or know in advance about the content of the critical scenario; otherwise, the interviewee will lose spontaneity.

(c) Pilot test and training - Both the scenario and the proposed questions should be overlooked by at least three researchers who have already carried out the CIT (Flanagan, 1973), in order to minimize the risks of errors in the data collection instruments and in the application of the technique. The pilot interview should also be applied to a professional with relevant experience in the studied field, in order to guarantee the validity and reliability of the results. At the same time, the pilot test serves as practical training on how to conduct the interviews and usually generates a series of changes in the critical scenario. Maximum attention should be given to comments received, especially negative ones. It is important to guarantee the confidentiality of the responses and the anonymity of the interviewees so that they do not feel embarrassed. It is expected, at least, that five people will participate in this pilot test. As a way of exemplifying, Figure 10 presents the critical scenario presented to the interviewees in Marchisotti’s (2021) research, which sought to find out the practical gaps on the topic of corporate governance.

Figure 10
Example of a critical scenario for data collection.

Once in possession of the critical scenario to be used for data collection, it will be possible to move on to the next phase, the interview.

Phase 2 - Interviews

This phase is the most important of this research stage, as the correct execution of the interviews guarantees quality material for further analysis. Therefore, in possession of the critical scenario, the researcher will go to the field to carry out the recorded interviews, which will later be transcribed to allow an appropriate analysis of their content. As a reference, at least 15 experts - who have reputable knowledge in the researched area or topic, dominating it to the point of exercising with full competence the knowledge they have developed - should be interviewed, including business professional or experts. It is important to mention that researchers must justify the number of interviews that were effectively executed, showing that the interviews’ numbers are enough to achieve the research’s objective (Saunders & Townsend, 2016). For this purpose, it is necessary to carry out five activities:

• (a) Selection and access to interviewees - Access to interviewees can be done through accessibility (Santos, Silva, Aguiar, Araújo, & Araújo, 2018) within the researcher’s contacts network; or through contacts obtained through LinkedIn or class associations, whose profile would demonstrate consistent practical experience in the researched topic. A minimum experience of five years is advisable, and the greater the differences in the characteristics of the interviewees - gender, race, sector, level of performance, among others -, the better return will be achieved.

• (b) Obtaining authorization for the interviews - Ethical standards need to be respected, so a free and informed consent term must be presented and signed by interviewees about their participation in the research, after understanding its objectives, including the guarantee of anonymity. The companies in which the interviewers work also must sign a term authorizing researcher to make interview with their employees.

• (c) Scheduling - All interviews should be scheduled in advance, by phone and virtual agendas - Google Calendar, Microsoft Outlook, or similar -, with the sending of a reminder of the interview to the interviewee the day before the scheduled date. If it is in person, it must be punctual; if it is remote, prepare in advance the software that will be used, such as Zoom or Skype application, for example.

• (d) Conducting the interviews - The interviews should be conducted in a quiet environment, without noise, and with maximum freedom for the interviewee, guaranteeing confidentiality from both the interviewed and the respective companies where they work. Before starting the interviews, the interviewee must be contextualized regarding the objectives of this phase of the work and the importance of sincere collaboration. Before introducing de critical scenario, do not forget to start recording the interview. It is also important to ask interviewers to make a brief summary of personal resumes, with an emphasis on experiences associated with the research topic. After that, introduce the critical scenario with stimulus questions, providing time for the interviewee to read it carefully, and then the respondent must answer the stimulus questions. All the audios of the interviews should be saved without identifying the names - Interviewee 1, Interviewee 2, etc., in order to guarantee the interviewees will not be identified by name to those who will transcribe the audios from the interviews. The interviews are expected to last at least 30 minutes to be able to collect useful data.

• (e) Transcription of the interview audios - All interview audios should be transcribed in text format, preferably by specialized professionals, so that their content can be analyzed by the NVIVO 11 software (Nvivo, 2017).

Thus, at the end of this phase, all interview data will be ready to have their content analyzed, in search of practical gaps.

Phase 3 - Consolidation

This phase refers to the consolidation of the collected data so that it is possible to analyze it. For this purpose, it is necessary to perform one activity:

(a) Content analysis - Perform the analysis of the content - codes and categories - of the interviews, with the support of the NVIVO 11 software (Nvivo, 2017), in order to be able to understand the meaning of what was expressed by the interviewees. As in the case of identifying theoretical gaps, a minimum of 15 and a maximum of 30 categories are expected, and a minimum of 50 and a maximum of 300 codes.

To identify practical gaps, it is recommended to carry out at least four coding cycles. In the first cycle, the initial codes generated and grouped according to the stimulus question from the critical scenario, according to the example used in this paper, are: advantages, disadvantages, failure factors, and success factors. This first codification is relevant so that it is possible to identify more clearly what the problems related to the researched theme would be, since the knowledge of the disadvantages and the success factors, mainly, can provide inputs to identify these problems, as well as the identification of their control variables and the relationships between them - hypotheses.

In a second coding cycle, new codes and subcodes are created and associated with content not directly associated with the first coding cycle - stimulus question, but with special attention to issues related to the researched topic. Then, a third coding cycle takes place, where all codes are reviewed, eliminating redundancies and similarities, and verifying that the content is correctly distributed in the different codes created.

In the fourth cycle, the researcher should group all codes associated with the topic: disadvantages and weaknesses of the researched theme. The researcher should give priority to creating the categories based on the codes associated with this topic, as this is where the problems or points of attention identified in the market practice, by the interviewees, will be better exposed. Thus, repeated codes or similar content should be grouped together, and the rest should be revised, allowing a view of possible categories and subcategories that are more adherent, according to Table 4.

Table 4
Example of content analysis based on the content analysis of the interviews - ten first categories.

Gap number	Most frequent words	Category inferred	Meaning
1	Public	Dominant public service	Public policies need to be transformed into public services, which are implemented by public companies through projects.
2	Service/ Services		The new approaches to the public employees that should be oriented to service provision and meeting the needs and wishes of end users.
3	Governance	Good governance	New demand in the way in which public management should behave, generating good public governance through appropriate control mechanisms.
4	Management	Performance management	New public management focused on managing the company and employees performance.
5	Performance
6	Government	NA	There were no expressions that stood out, since the word government was used in various ways and in different contexts.
7	Project / Projects	Project management	The correct Project management in the public sector increases the implementation rate of the company’s strategy, with potential performance improvement.
8	Political	Political activity	Decision-making in public companies must be independent of political interference when it is harmful. The focus should be on improving performance and service delivery and not serving political interests.
9	Sector	Public partnerships	The public sector needs to build bridges with other sectors, improving performance and service quality
10	Private		There is a need to make public-private partnerships, combined resources, cultures, values and competence, since public services are more modern and with more complex problems

• Note. Source: Marchisotti (2021).

As a way to guarantee the reliability of the results, both after transcribing the interviews and at the end of the content analysis, the researcher should validate and legitimize all the material with the interviewees, mitigating errors of interpretation or biases of the researcher and transcriber (Fraser & Gondim, 2004).

It is worth mentioning that in the identification of practical gaps there is a chance of using grounded theory over content analysis, following the theoretical framework of Corbin and Strauss (2008). In this case, it is necessary, during coding and categorizing the content, to use the same words and form as mentioned by the interviewees, with no space for the researcher to manipulate them (Andrews, Mariano, Santos, Koerber-Timmons, & Silva, 2017).

At the end of this Step 2, it is possible to identify a series of practical gaps or points of attention, based on the findings of the analysis content of the expert interviews, which can be used as research problems.

Stage 3 - Theoretical-practical gaps

Finally, for the identification of the theoretical-practical gaps, that is, the gaps that were addressed by both theory and experts, the results of the content analysis of Step 1 are compared with those of Step 2, according to Figure 11.

Figure 11
Identification of theoretical-practical gaps.

Each of the categories generated - and their respective associated problems - must be compared, in search of items converging in their meaning and scope. Thus, from the comparison between the practical and theoretical gaps - Tables 3 and 4 - it is possible to identify gaps that are present both in theory and in practice - theoretical-practical gaps, according to Table 5.

Table 5
Example of the comparison result of the first ten practical and theoretical gaps, generating the theoretical-practical gaps.

Gaps number	Theoretical gap	Practical gap	Theoretical-practical gap
1	Gap number 8 - Political activity	Gap number 3 - Harmful political influence	Political influences are potentially harmful to the governance system and the state-owned company´s results. How to shield the company?

• Note. Source: Marchisotti (2021).

At the end of this stage, it will be possible to identify research problems based on theory, or on practice, or on theory and practice at the same time, so that the researcher can choose any of them as his research problem. The ideal would be the use of a problem originating from the theoretical-practical gap seeking to integrate different theories as a way of solving it, as proposed by Hällgren (2012).

CONCLUSION

The paper achieves its objective of detailing the tutorial to be followed by researchers and students, in order to operationalize the theoretical multi-method proposal to identify a research problem put forward by Farias et al. (2019). The present paper shows in detail an alternative to be followed by the researchers to find their research problem, without invalidating, however, other possibilities of executing this multi-method in search of the research problem identification.

It is concluded that it is a robust procedure associating the methods of identifying research problems based on theoretical gaps and practical need or examples, as approached by Sandberg and Alvesson (2011), Hällgren (2012), and Okimoto (2014). Integrating different methods to identifying the research problem is, therefore, more complete than the separate use of each of them, going beyond the usual identification of the research problem based only on theoretical gaps.

It is worth mentioning that all the phases proposed in this paper need to be followed methodically, from the choice of a broad academic database, passing through the choice of experts who can effectively contribute to the topic to be researched, until content analysis of the content of all collected data. It would be useless to collect data properly if the analysis of the researcher is influenced by biases or errors during the analysis process, which despite being subjective, can be performed in a procedural manner, mitigating the risks of incorrect analysis.

An important academic contribution of this work is the probability of instrumentation of researchers, with a progressive procedure to be followed, enabling the implementation of the multi-method proposed by Farias et al. (2019) in a proper way. Thus, the doubts regarding the use of this research method are resolved, especially regarding its practical operationalization, since this is usually a big concern and difficulty for students and researchers. It should be adopted by undergraduate and graduate students because it proves to be complete, efficient, and effective in identifying analytically relevant research questions.

One of the limitations of this research is that it does not describe the grounded theory procedures that can be used instead of content analyses. Although the step-by-step procedures described in this paper have been adjusted over the years during which they have been used by researchers, they have not been used in all areas of knowledge, so it is not guaranteed that they will work for all. This procedure must be constantly and dynamically updated, especially considering the use of different technologies as artificial intelligence or similar, which could be integrated into the procedure.

It is suggested that future studies define a progressive approach for using established grounded theory in place of content analysis of practical gaps, so that it is possible to be even closer to reality, making use of the emerging data more closely to the reality presented by the field. It is also suggested that other forms of research problem identification associated with problematization be tested, considering its different available typologies, in order to challenge the current theoretical assumptions more intensely.

The step-by-step method proposed in this paper can be used in several projects, in different areas of knowledge, in order to make it even more robust and consistent, after the practical feedback and real applications by other researchers.

REFERENCES

• Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247-271. https://doi.org/10.5465/amr.2009.0188
  » https://doi.org/10.5465/amr.2009.0188
• Alvesson, M., & Sandberg, J. (2013). Constructing research questions: Doing interesting research. London: Sage Publications.
• Andrews, T., Mariano, G. J. D. S., Santos, J. L. G. D., Koerber-Timmons, K., & Silva, F. H. D. (2017). A metodologia da teoria fundamentada nos dados clássica: Considerações sobre sua aplicação na pesquisa em enfermagem. Texto & Contexto-Enfermagem, 26(4). https://doi.org/10.1590/0104-070720170001560017
  » https://doi.org/10.1590/0104-070720170001560017
• Braga, J. L. (2005). Para começar um projeto de pesquisa. Comunicação & Educação, 10(3), 288-296. https://doi.org/10.11606/issn.2316-9125.v10i3p288-296
  » https://doi.org/10.11606/issn.2316-9125.v10i3p288-296
• Clarivate (2020). Incites JCR. Retrieved from https://jcr.clarivate.com/
  » https://jcr.clarivate.com/
• Cooper, D. R., & Schindler, P. S. (2016). Métodos de Pesquisa em Administração (12^nd ed.) Porto Alegre: AMGH.
• Corbin, J., & Strauss, A. (2008). Strategies for qualitative data analysis. In Basics of qualitative research: Techniques and procedures for developing grounded theory (vol. 3). Los Angeles: Sage.
• Creswell, J. W. (2016). Projeto de pesquisa: Métodos qualitativo, quantitativo e misto (2^nd ed.) Porto Alegre: Penso.
• David, M. (2017). Statistics for managers, using Microsoft excel (8^th ed.). London: Pearson.
• Farias, J. R. Filho, Marchisotti, G. G., Maggessi, K. M. F., & Miranda, H. L. de, Junior. (2019). Método de pesquisa misto para identificação do problema de pesquisa. Conhecimento & Diversidade, 10(22), 88-102. http://dx.doi.org/10.18316/rcd.v10i22.5155
  » http://dx.doi.org/10.18316/rcd.v10i22.5155
• Fernandes, A. T. (2017). Alguns desafios teórico-metodológicos. Sociologia: Revista da Faculdade de Letras da Universidade do Porto, 6. Retrieved from from https://ojs.letras.up.pt/index.php/Sociologia/article/viewFile/2592/2376
  » https://ojs.letras.up.pt/index.php/Sociologia/article/viewFile/2592/2376
• Flanagan, J. C. (1973). A técnica do incidente crítico. Arquivos Brasileiros de Psicologia Aplicada, 25(2), 99-141. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/abpa/article/download/16975/15786
  » http://bibliotecadigital.fgv.br/ojs/index.php/abpa/article/download/16975/15786
• Fraser, M. T. D., & Gondim, S. M. G. (2004). Da fala do outro ao texto negociado: Discussões sobre a entrevista na pesquisa qualitativa. Paidéia (Ribeirão Preto), 14(28), 139-152. https://doi.org/10.1590/S0103-863X2004000200004
  » https://doi.org/10.1590/S0103-863X2004000200004
• Gray, D. E. (2016). Pesquisa no mundo real (2^nd ed.). Porto Alegre: Penso.
• Hatch, J. A. (2002). Doing qualitative research in education settings. New York: Suny Press.
• Hällgren, M. (2012). The construction of research questions in project management. International Journal of Project Management, 30(7), 804-816. httpS://doi.org/10.1016/j.ijproman.2012.01.005
  » httpS://doi.org/10.1016/j.ijproman.2012.01.005
• Hupe, M. (2019). EndNote X9. Journal of Electronic Resources in Medical Libraries, 16(3-4), 117-119. https://doi.org/10.1080/15424065.2019.1691963
  » https://doi.org/10.1080/15424065.2019.1691963
• Lana, J., Partyka, R. B., Alberton, A., & Marcon, R. (2018). Caso para ensino: O processo de escolhas metodológicas em uma abordagem quantitativa. Revista de Contabilidade e Organizações, 12, e148286. https://doi.org/10.11606/issn.1982-6486.rco.2018.148286
  » https://doi.org/10.11606/issn.1982-6486.rco.2018.148286
• Lacerda, R. T. D. O., Ensslin, L., & Ensslin, S. R. (2012). Uma análise bibliométrica da literatura sobre estratégia e avaliação de desempenho. Gestão & Produção, 19, 59-78. Retrieved from https://www.scielo.br/j/gp/a/sKh5wfCCGv68fdRP8GStLXC/?lang=pt&format=pdf
  » https://www.scielo.br/j/gp/a/sKh5wfCCGv68fdRP8GStLXC/?lang=pt&format=pdf
• Lordelo, S., A., V. (2019). Modelo multicritério para escolha de ações para a redução do retrabalho em empreendimentos complexos (Doctoral thesis). Universidade Federal Fluminense, Niterói, RJ, Brazil.
• Marchisotti, G. G. (2021). Modelo para avaliação da percepção de valor de um sistema de governança das organizações (Doctoral thesis). Universidade Federal Fluminense, Niterói, RJ, Brazil.
• Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & López-Cózar, E. D. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160-1177. https://doi.org/10.1016/j.joi.2018.09.002
  » https://doi.org/10.1016/j.joi.2018.09.002
• Moraes, C. S. B. de, Vale, N. P. do, & Araújo, J. A. (2013). Sistema de Gestão Integrado (SGI) e os benefícios para o setor siderúrgico. Revista Metropolitana de Sustentabilidade, 3(3), 29-48. Retrieved from https://revistaseletronicas.fmu.br/index.php/rms/article/view/214
  » https://revistaseletronicas.fmu.br/index.php/rms/article/view/214
• Morgado, F. (2017). Microsoft Word Secrets: The why and how of getting word to do what you want. New York: Apress.
• Nvivo (2017). Nvivo 11 Pro for Windows. QSR International Pty Ltd. Recovered from http://download.qsrinternational.com/Document/NVivo11/11.4.0/pt-BR/NVivo11-Getting-Started-Guide-Pro-edition-Portuguese.pdf.
  » http://download.qsrinternational.com/Document/NVivo11/11.4.0/pt-BR/NVivo11-Getting-Started-Guide-Pro-edition-Portuguese.pdf.
• Okimoto, T. G. (2014). Toward more interesting research questions: Problematizing theory in social justice. Social Justice Research, 27(3), 395-411. https://doi.org/10.1007/s11211-014-0215-5
  » https://doi.org/10.1007/s11211-014-0215-5
• Oliveira, J. L. C. D., Magalhães, A. M. M. D., & MisueMatsuda, L. (2018). Métodos mistos na pesquisa em enfermagem: Possibilidades de aplicação à luz de Creswell. Texto & Contexto-Enfermagem, 27(2). https://doi.org/10.1590/0104-070720180000560017
  » https://doi.org/10.1590/0104-070720180000560017
• Oliveira, J. T. de, & Zanotti, A. (2018). A Importância do raciocínio lógico para a metodologia da pesquisa: Raciocínio, planejamento e execução. Research, Society and Development, 7(8), e978337. https://doi.org/10.17648/rsd-v7i8.337
  » https://doi.org/10.17648/rsd-v7i8.337
• Padilha, M. I. C. D. S., & Borenstein, M. S. (2005). O método de pesquisa histórica na enfermagem. Texto & Contexto-Enfermagem, 14, 575-584. https://doi.org/10.1590/S0104-07072005000400015
  » https://doi.org/10.1590/S0104-07072005000400015
• Paranhos, R., Figueiredo, D. B. Filho, Rocha, E. C. D., Silva, J. A. D. Júnior, & Freitas, D. (2016). Uma introdução aos métodos mistos. Sociologias, 18(42), 384-411. https://doi.org/10.1590/15174522-018004221
  » https://doi.org/10.1590/15174522-018004221
• Quelhas, A. D. (2017) Pesquisa mista para proposição de um modelo para analisar a relevância do tripé cultura, clima e comportamento organizacional no desempenho empresarial e mensuração do nível de aderência numa construtora na região metropolitana do Rio de Janeiro (Doctoral thesis). Universidade Federal Fluminense, Niterói, RJ, Brazil.
• Saldaña, J. (2015). The coding manual for qualitative researchers (3^rd ed.). London: Sage.
• Sandberg, J., & Alvesson, M. (2011). Ways of constructing research questions: Gap-spotting or problematization? Organization, 18(1), 23-44. https://doi.org/10.1177/1350508410372151
  » https://doi.org/10.1177/1350508410372151
• Santos, A. A. dos, Silva, M. das G., Aguiar, S. G. de, Araújo, J. G. N. de, & Araújo, J. G. de. (2018). A percepção do profissional contábil no processo de convergência nas pequenas e médias empresas na região metropolitana do Recife-PE. Revista UNEMAT de Contabilidade, 7(13). https://doi.org/10.30681/ruc.v7i13.1652
  » https://doi.org/10.30681/ruc.v7i13.1652
• Saunders, M. N., & Townsend, K. (2016). Reporting and justifying the number of interview participants in organization and workplace research. British Journal of Management, 27(4), 836-852. https://doi.org/10.1111/1467-8551.12182
  » https://doi.org/10.1111/1467-8551.12182
• Scopus (2018). Welcome to Scopus Preview. Retrieved from https://www.scopus.com/home.uri
  » https://www.scopus.com/home.uri
• Silva, D. D., & Grácio, M. C. C. (2017). Índice h de Hirsch: Análise comparativa entre as bases de dados Scopus, Web of Science e Google Acadêmico. Em Questão, 23, 196-212. https://doi.org/10.19132/1808-5245230.196-212
  » https://doi.org/10.19132/1808-5245230.196-212
• Tani, G. (1992). Contribuições da aprendizagem motora à educação física: Uma análise crítica. Revista Paulista de Educação Física, 6(2), 65-72. https://doi.org/10.11606/issn.2594-5904.rpef.1992.138073
  » https://doi.org/10.11606/issn.2594-5904.rpef.1992.138073
• Toledo, R. (2020). Modelo para integrar sustentabilidade na gestão de projetos (Doctoral thesis). Universidade Federal Fluminense, Niterói, RJ, Brasil.
• Treinta, F. T., Farias, J. R., Sant’Anna, A. P., & Rabelo, L. M. (2014). Metodologia de pesquisa bibliográfica com a utilização de método multicritério de apoio à decisão. Production, 24(3), 508-520. https://doi.org/10.1590/S0103-65132013005000078
  » https://doi.org/10.1590/S0103-65132013005000078
• Tripp, D. (2005). Pesquisa-ação: Uma introdução metodológica. Educação e pesquisa, 31(3), 443-466. http://dx.doi.org/10.1590/S1517-97022005000300009
  » http://dx.doi.org/10.1590/S1517-97022005000300009
• Zappellini, M. B., & Feuerschütte, S. G. (2015). O uso da triangulação na pesquisa científica brasileira em administração. Administração: Ensino e Pesquisa, 16(2), 241-273. http://dx.doi.org/10.13058/raep.2015.v 16n2.238
  » http://dx.doi.org/10.13058/raep.2015.v 16n2.238

• JEL Code:

  C00, M00.

• Peer Review Report:

  The Peer Review Report is available at this external URL.

• Funding

  The authors reported that there is no financial support for the research in this article.

• Copyrights

  RAC owns the copyright to this content.

• Plagiarism Check

  The RAC maintains the practice of submitting all documents approved for publication to the plagiarism check, using specific tools, e.g.: iThenticate.

• Peer Review Method

  This content was evaluated using the double-blind peer review process. The disclosure of the reviewers’ information on the first page, as well as the Peer Review Report, is made only after concluding the evaluation process, and with the voluntary consent of the respective reviewers and authors.

• Data Availability

  RAC encourages data sharing but, in compliance with ethical principles, it does not demand the disclosure of any means of identifying research subjects, preserving the privacy of research subjects. The practice of open data is to enable the reproducibility of results, and to ensure the unrestricted transparency of the results of the published research, without requiring the identity of research subjects.

APPENDIX A - Example of calculating the papers’ final score

Table A1
Example of calculating the papers final score according to the CPP method.

Title	Authors	Academic Journal	Keywords	Abstract	Year	Citation	Article Authorship (h-index of first author)	Academic Journal Evaluation		Thematic Alignment	Final Score
						Weight = 0.23	Weight = 0.13	SJR	SNIP	Weight = 0.42
								Weight = 0.22
The three dimensions of a governance framework for major public projects	Brunet, Maude and Monique Aubry	International Journal of Project Management	Governance framework, Major public projects, Public administration, Project governance, Conceptual article.	The aim of this article is to advance a conceptualization for governance frameworks for major public projects based on public administration literature. The governance of major public projects has been an important subject of inquiry in project management, as researchers have investigated governance frameworks for public projects as a tool to enhance performance. Yet, while performance is traditionally seen as improved efficiency, other aspects need consideration. Using phronetic and abductive theory building, this conceptual article investigates the relevance of a governance framework for major public projects along three dimensions: those of greater government efficiency, legitimacy and accountability. The main contribution of this article is to enrich existing theory on the governance of major public projects.	2016	63	5	2.659	3.561	9	= [(7*0.23) + (3*0.13) + (6*0.22) + (9*0.42)] = 1.61 + 0.39 + 1.32 + 3.78 = 7.1
						7*	3*	5*	7*
								= (5+7)/2 = 6

• Note. * Converted using scale 1 to 9 - 1, 3, 5, 7, and 9, considering the minimum and maximum values ​of all papers analyzed in the sample. Source: Adapted from Marchisotti (2021).

Edited by

Editors-in-chief:

Wesley Mendes-da-Silva (Fundação Getulio Vargas, EAESP, Brazil)

Marcelo de Souza Bispo (Universidade Federal da Paraíba, PPGA, Brazil)

Associate Editor:

Evelyn Lanka (Cranfield School of Management, United Kingdom)

Publication Dates

• Publication in this collection
  18 July 2022
• Date of issue
  2022

History

• Received
  18 Jan 2021
• Reviewed
  15 Oct 2021
• Accepted
  22 Nov 2021