# Abstract

Keywords
Research expertise; Research capability; Evidence-based analysis; Academic relationship; Industrial engineering education

# 1 Introduction

In recent decades, investments in scientific capacity have been viewed as a critical precursor for creating economic growth in the knowledge economy by governments worldwide (Feldman et al., 2014Feldman, M. P., Lanahan, L., & Lendel, I. V. (2014). Experiments in the laboratories of democracy: state scientific capacity building. Economic Development Quarterly, 28(2), 107-131. http://dx.doi.org/10.1177/0891242413490018.
http://dx.doi.org/10.1177/08912424134900...
). However, because of the limited public budgets and increased competition during the research funding applications, numerous academic faculties have been focused on finding some ways to improve their research capability (Smeby & Try, 2005Smeby, J. C., & Try, S. (2005). Departmental contexts and faculty research activity in Norway. Research in Higher Education, 46(6), 593-619. http://dx.doi.org/10.1007/s11162-004-4136-2.
http://dx.doi.org/10.1007/s11162-004-413...
). In order to support the academic faculty development, the investigation of the existing issues in the current research activities is usually considered as the primary and challenging step. Those research activities are influenced by many factors, such as personalized characteristics and specific knowledge background of academic staffs, and also the diverse resources that are available for researches (Adler et al., 2009Adler, N., Elmquist, M., & Norrgren, F. (2009). The challenge of managing boundary-spanning research activities: experiences from the Swedish context. Research Policy, 38(7), 1136-1149. http://dx.doi.org/10.1016/j.respol.2009.05.001.
http://dx.doi.org/10.1016/j.respol.2009....
). As the output of research activities, academic articles are often considered as the approved and the evidence-based resource for this kind of investigation (Arundel & Constantelou, 2006Arundel, A., & Constantelou, A. (2006). Conventional and experimental indicators of knowledge flows. In Y. Caloghirou, A. Constantelou & N. Vonortas (Eds.), Knowledge flows in European industry (pp. 45-66). London: Routledge. Retrieved in 21 July 2016, from http://ecite.utas.edu.au/75683
http://ecite.utas.edu.au/75683...
).

Along with the fast growth in the number of publications, an increasing variety of article review and evaluation methods are proposed. In general, these methods can be divided into two kinds: (1) the systematic review methods that focus on investigating the contents in the articles, such as the rapid review (Khangura et al., 2012Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 1(1), 10. PMid:22587960. http://dx.doi.org/10.1186/2046-4053-1-10.
http://dx.doi.org/10.1186/2046-4053-1-10...
), the scoping study (Arksey & O’Malley, 2005Arksey, H., & O’Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. http://dx.doi.org/10.1080/1364557032000119616.
http://dx.doi.org/10.1080/13645570320001...
), and the meta-analysis (Mullen, 1986Mullen, B. (1986). Basic meta-analysis: description of a statistical package. Behavior Research Methods, Instruments, & Computers, 18(2), 165-167. http://dx.doi.org/10.3758/BF03201018.
http://dx.doi.org/10.3758/BF03201018...
); (2) the research productivity evaluation methods that based on some pre-defined criteria, such as volume, quality, impact, and utility (King, 1987King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of Information Science, 13(5), 261-276. http://dx.doi.org/10.1177/016555158701300501.
http://dx.doi.org/10.1177/01655515870130...
; Geuna & Martin, 2003Geuna, A., & Martin, B. R. (2003). University research evaluation and funding: an international comparison. Minerva, 41(4), 277-304. http://dx.doi.org/10.1023/B:MINE.0000005155.70870.bd.
http://dx.doi.org/10.1023/B:MINE.0000005...
; Higher Education Funding Council for England, 2009Higher Education Funding Council for England. (2009). Research Excellence framework: second consultation on the assessment and funding of research. Bristol: HEFCE. Retrieved in 21 July 2016, from http://www.hefce.ac.uk/pubs/year/2009/200938/
http://www.hefce.ac.uk/pubs/year/2009/20...
). For the first kind, it shows the capability of analysis the “inside view”, namely the summary of the known and unknown knowledge that relates to a specific research topic. For the second kind, despite there is still debate (Smith et al., 2011Smith, S., Ward, V., & House, A. (2011). Impact’ in the proposals for the UK’s research excellence framework: shifting the boundaries of academic autonomy. Research Policy, 40(10), 1369-1379. http://dx.doi.org/10.1016/j.respol.2011.05.026.
http://dx.doi.org/10.1016/j.respol.2011....
), it focuses more in the “outside view”, namely the research production assessment of one or more academic staffs.

As pointed out by Bercovitz & Feldman (2011)Bercovitz, J., & Feldman, M. (2011). The mechanisms of collaboration in inventive teams: composition, social networks, and geography. Research Policy, 40(1), 81-93. http://dx.doi.org/10.1016/j.respol.2010.09.008.
http://dx.doi.org/10.1016/j.respol.2010....
, research activities are generally carried out in teams, which bring together numerous academic staffs from different knowledge domains and organizations. We found despite the big methodological efforts have been made by above-mentioned article review methods, less attention has been paid to the discovery of the implicit academic relationships among the academic staffs and neither the investigation of their research expertise. Several researchers (He et al., 2009He, Z.-L., Geng, X.-S., & Campbell-Hunt, C. (2009). Research collaboration and research output: a longitudinal study of 65 biomedical scientists in a New Zealand university. Research Policy, 38(2), 306-317. http://dx.doi.org/10.1016/j.respol.2008.11.011.
http://dx.doi.org/10.1016/j.respol.2008....
; Hoekman et al., 2010Hoekman, J., Frenken, K., & Tijssen, R. J. W. (2010). Research collaboration at a distance: changing spatial patterns of scientific collaboration within Europe. Research Policy, 39(5), 662-673. http://dx.doi.org/10.1016/j.respol.2010.01.012.
http://dx.doi.org/10.1016/j.respol.2010....
; Li et al., 2014Li, F., Miao, Y., & Yang, C. (2014). How do alumni faculty behave in research collaboration? An analysis of Chang Jiang Scholars in China. Research Policy, 44(2), 438-450. http://dx.doi.org/10.1016/j.respol.2014.09.002.
http://dx.doi.org/10.1016/j.respol.2014....
) have begun to realize this issue and started to discover its corresponding solution. However, these existing studies were mainly designed for analyzing the collaboration issues in some special example of cases, where there is still a lack of a general method that can be easily adopted.

The objective of this paper is to address this existing drawback through the proposition of an Academic Information Classification and Summarization method. The rest of this paper is organized as follows: Section 2 presents an academic information classification table and a formalized procedure for the summarization. Based on the proposed method, Section 3 illustrates a case study to show the advantages that can be obtained from the proposition. Section 4 concludes this paper and highlights further works.

# 2 Academic information classification and summarization method

This section presents a gradual strategy. It firstly classifies the academic information from the collected academic articles, and then it uses this result to generate summaries from different perspectives within an academic faculty. An overview of the procedure is given in Figure 1. This workflow is divided into three main phases: the collection phase, the classification phase, and the summarization phase.

Figure 1
The workflow of the academic information classification and summarization method.

## 2.1 The collection phase

During this phase, through the definition and application of the article selection parameters, the academic articles that are needed by the next phase (the classification phase) are collected. Six main parameters, which we used in this paper, are introduced as follows:

• The Name of the Academic Faculty”: the academic faculty that has the intention to analyze the research activities within it;

• The Selected Article Database”: the commonly approved database, which can cover the main research output (academic articles) in that academic faculty;

• The Range of Years”: the publication period of those academic articles;

• The Publication Languages”: the languages of those academic articles;

• The Publication Types”: the types of those academic articles, such as journal articles, and conference papers.

Of course not only limited to these preliminary parameters, the article selection parameters can be extended from both the completeness aspect and the specificity aspect. For the former aspect, more new parameters can be employed. For example, adding a “The Subject Areas” parameter to limit the domains of those academic articles. For the latter aspect, more restrictions of the existing parameters can be further specified. For example, refining the “The Publication Types” with a restriction that limits the article selection within one or more appointed journals.

## 2.2 The classification phase

The full text review of academic articles is a time consuming work, even if the analysis boundary within a research group. Due to the keywords usually can cover most of important concepts that included in an academic article (Chen et al., 2008Chen, N.-S., Wei, C.-W., & Chen, H.-J. (2008). Mining e-Learning domain concept map from academic articles. Comput. Educ., 50(3), 1009-1021. http://dx.doi.org/10.1016/j.compedu.2006.10.001.
http://dx.doi.org/10.1016/j.compedu.2006...
), we propose to firstly classify the keywords, and then use them to construct a topic sentence that can represent the essence of an academic article.

As can be seen from Table 1, each column in that table shows one aspect of academic information that is extracted and classified from an article. To be more specific, the semantics of each column are introduced as follows:

Table 1
• No.”: the identifier of an academic article;

• When”: the publication year of that academic article;

• Who”: the author names of that academic article;

• What”: the objects of study in that academic article;

• How”: the methods that are applied on the “What” to deal with issues about the “Why”;

• Where”: the application contexts of the “How”, “What” and “Why”.

In order to capture the essence of an academic article and also reconfirm the correctness of preliminary classification, a topic sentence construction formula is proposed as follow: “In [When], [Who] applied the [How] on the [What] to deal with issues about the [Why] in the context of the [Where] ”.

The contents in each row are firstly filled with the existing or extracted keywords from an academic article. And then the formula takes the classification results from the table and constructs a corresponding topic sentence for each collected article. In some specific cases, publications might not be able to provide an appropriate list of keywords to express their topics. In this paper, this kind of insufficient situation means that the keywords of an article are unable to fulfill all the grids in a row of the classification table. In this case, for the empty grids, the corresponding keywords need to be extracted from the title, the abstract, or even the full text of that academic article before the classification. Because a variety of keyword extraction methods have already been proposed by many researchers, such as Hulth (2003)Hulth, A. (2003). Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (Vol. 10, pp. 216-223). Stroudsburg, USA. http://dx.doi.org/10.3115/1119355.1119383
http://dx.doi.org/10.3115/1119355.111938...
, Matsuo & Ishizuka (2004)Matsuo, Y., & Ishizuka, M. (2004). Keyword extraction from a single document using word co-occurrence statistical information. International Journal of Artificial Intelligence Tools, 13(1), 157-169. http://dx.doi.org/10.1142/S0218213004001466.
http://dx.doi.org/10.1142/S0218213004001...
and Yih et al. (2006)Yih, W., Goodman, J., & Carvalho, V. R. (2006). Finding advertising keywords on web pages. In Proceedings of the 15th International Conference on World Wide Web (WWW ’06) (pp. 213-222), Edinburgh, Scotland. http://dx.doi.org/10.1145/1135777.1135813.
http://dx.doi.org/10.1145/1135777.113581...
, the development of the related technology for the keyword extraction is not the focus of this paper.

## 2.3 The summarization phase

During this phase, the investigation of the scientific research activities in an academic faculty is performed based on the academic information classification table (the output of the classification phase). The contents in each row of the table show the five main perspective (“When”, “What”, “Why”, “How”, and “Where”) of an article (“No.”) that is contributed by one or more academic staffs (“Who”). Through the combination of these seven elements, various kinds of summaries can be exhibited differently. As examples in this paper, we list five combinations as follows:

• No.” & “Who” & “When”. This combination shows the collaboration frequency in different years (“When”) among the academic staffs (“Who”) in an academic faculty;

• No.” & “Who” & “What”. This combination shows the main research objects (“What”) that the academic staffs (“Who”) worked on;

• No.” & “Who” & “Why”. This combination shows the main research problems (“Why”) that the academic staffs (“Who”) dealt with;

• No.” & “Who” & “How”. This combination shows the main methods or technologies (“How”) that the academic staffs (“Who”) can handle;

• No.” & “Who” & “Where”. This combination shows the main research domains (“Where”) that the academic staffs (“Who”) took part in.

Let $N$ be the set of the article identifiers that are described by the “No.” in the classification table, and $n∈N$ be the article identifier in row $n$. Let $P$ be the set of academic staffs in an academic faculty, and $Pn⊆P$ be the set of academic staffs that are described by the “Who” in row $n$. In the remaining of this section, we present two formal summarization methods: (1) the academic relationship summarization, and (2) the research expertise summarization.

### 2.3.1 The academic relationship summarization

The summaries of the academic relationships among the academic staffs in an academic faculty can be based on the combination of the “No.” & “Who” & “When” (from the time and frequency perspective) and the combination of the “No.” & “Who” & “Where” (from the domain and frequency perspective), are presented as follows.

For the first combination, let $Y$ be the set of years that are described by the “When” in the classification table, and $yn∈Y$ be the year in row $n$. In order to summarize the collaboration among those academic staffs from the time and frequency perspective, the figure is suggested to satisfy the following conditions:

• Each academic staff in $P$ is represented as an entity in the figure;

• A no repeat relation $rij=(pi,pj)$ is created between $pi∈P$ and $pj∈P$, where $pi≠pj$, and $pi, pj∈Pn$;

• A set of years $Yij∈Y$, which records the years of collaboration between $pi∈P$ and $pj∈P$ is attached to the relation $rij=(pi,pj)$, where $pi≠pj$, $pi, pj∈Pn$, and $yn∈Yij$;

• For each element in $Yij$, a frequency number $fij$, which records the number of repetitions of $pi∈P$, $pj∈P$, and $yn∈Yij$ that all appear in a same row $n$, is attached to it.

For the second combination, a generally accepted domain framework should be firstly employed as its background. In that framework, different research domains are placed at different positions, together with a certain form of relationships between each other. Let $RD$ be the set of research domains and $DP$ be the set of positions in the employed framework. Let $D$ be the set of research domains that are described by “Where” in the classification table, and $Dn∈D$ be the set research domains in row $n$. In order to provide an overview of the collaboration among those academic staffs from the domain and frequency perspective, the figure is suggested to satisfy the following conditions:

• For each research domain $rdx∈RD$ in the framework, there is a corresponding position $dpx∈DP$;

• For each position $dpx∈DP$, there is a set of academic staffs $Px⊆P$ that is attached to it;

• For each academic staff $pny∈Pn$ in row $n$, if a research domain $dnz∈Dn$ is equal to a research domain $rdx∈RD$, then $pny$ belongs to $Px$. That is $∀pny∈Pn, ∃dnz∈Dn=rdx∈RD ⇒ pny∈Px$

• For each academic staff $pxl∈Px$, there is a frequency number $fxl$, which records the number of repetitions of that academic staff $pxl$ and the researcher domain $rdx$ that both appear in a same row, is attached to it.

### 2.3.2 The research expertise summarization

The research expertise summarization of the academic staffs in an academic faculty can be based on the combination of the “No.” & “Who” & “What” (from the research object perspective), the combination of the “No.” & “Who” & “Why” (from the research challenge perspective), and the combination of the “No.” & “Who” & “How” (from the research method perspective). In this paper, we represent each of above-mentioned combinations through a set of pie charts respectively. Each summary represents an academic staff $pi∈P$ as an entity and is attached by its corresponding pie chart $pci$. Let $Si$ be the set of sectors in the pie chart $pci$.

For the first combination, let $O$ be the set of research objects that are described by “What” in the classification table, and $On⊆O$ be the set of research objects in row $n$. In order to show an overview of the research expertise from the research object perspective, the figure is suggested to satisfy the following conditions:

• Let $Oi⊆O$ be the set of research objects that is worked on by an academic staff $pi$. If one $pi$ and one research object $ont∈On$ that both appear in row $n$, then $ont∈Oi$;

• Each research object in $Oi$ is represented as a sector in $Si$;

• The numerical proportions of the sectors in $Si$ are based on the number of repetitions of the academic staff $pi$ and its corresponding research objects in $Oi$ that both appear in a same row.

For the second combination, let $C$ be the set of research challenges that are described by “Why” in the classification table, and $Cn⊆C$ be the subset of research challenges in row $n$. In order to show an overview of the research expertise from the research challenge perspective, the figure is suggested to satisfy the following conditions:

• Let $Ci⊆C$ be the set of research challenges that is dealt with by an academic staff $pi$. If a $pi$ and a research challenge $cnu∈Cn$ that both appear in row $n$, then $cnu∈Ci$;

• Each research challenge in $Ci$ is represented as a sector in $Si$;

• The numerical proportions of the sectors in $Si$ are based on the number of repetitions of the academic staff $pi$ and its corresponding research challenge in $Ci$ that both appear in a same row.

For the third combination, let $M$ be the set of research methods that are described by “How” in the classification table, and $Mn⊆M$ be the set of research methods in row $n$. In order to show an overview of the research expertise from the research method perspective, the figure is suggested to satisfy the following conditions:

• Let $Mi⊆M$ be the set of research methods that can be handled by an academic staff $pi$. If a $pi$ and a research method $mnk∈Mn$ that both appear in row $n$, then $mnk∈Mi$;

• Each research method in $Mi$ is represented as a sector in $Si$;

• The numerical proportions of the sectors in $Si$ are based on the number of repetitions of the academic staff $pi$ and its corresponding research method in $Mi$ that both appear in a same row.

After the summarization phase, an evidence-based result that outlines the scientific research activities of a research group from five different perspectives will be generated. Due to diverse academic faculties might have various kinds of evaluation or assessment criteria, the way to make conclusions based on the summarized results will be an object of continuous exploration.

# 3 The case study

In this section, a case study is taken place in the academic faculty named PPGEPS (Graduate Program in Production Engineering and Systems, Pontifical Catholic University of Paraná, Brazil) and is used to show the advantages that can be obtained from the proposed method. Based on the workflow of this method, this section is divided into three corresponding subsections.

## 3.1 Case study: collection phase

In this academic faculty, we would like to analyze the research activities of 14 academic staffs in the last three years. A well-accepted platform in Brazil, named “Cnpq Plataforma Lattes”, is chosen as the literature source. In this platform, all of the academic staffs in this faculty have already updated their publications in their resumes. As can be seen from Table 2, the details of the article selection parameters are shown.

Table 2
Article selection parameters in the case study.

At the end of the collection, totally 76 journal articles and 86 conference proceedings that satisfied these conditions are discovered. They are acting the basic resource for the classification phase.

## 3.2 Case study: classification phase

According to the proposed method, the keywords of the collected articles are studied, extracted and classified. And then, the preliminary classification results are used to construct the topic sentences and to reconfirm their correctness. As can be seen from Figure 2, one classification result of an example article (Liao et al., 2014Liao, Y., Lezoche, M., Loures, E. R., Panetto, H., & Boudjlida, N. (2014). Formal semantic annotations for models interoperability in a PLM environment. In Proceedings of the 19th World Congress of the International Federation of Automatic Control (IFAC 2014) (pp. 2382-2393), Cape Town, South Africa. http://dx.doi.org/10.3182/20140824-6-ZA-1003.02551
http://dx.doi.org/10.3182/20140824-6-ZA-...
) is shown.

Figure 2
The classified academic information from an example article.

Among these collected academic articles, we found that 75 of them do not contain sufficient keywords and 11 of them do not contain keywords, which request for a keyword extraction process. After the preliminary classification, the keywords in each row of this table are used to create a corresponding topic sentence for each collected academic article and to reconfirm their correctness. At the end, a classification table that contains 162 rows is generated. Together with the six parameters in the previous phase, they are used inputs for the summarization phase.

## 3.3 Case study: summarization phase

Based on five combinations and their corresponding summarization procedures, the summaries of the academic relationships and the research expertise are generated.

### 3.3.1 The Summaries of the Academic Relationships

As can be seen from Figure 3, the summary of the academic relationships among the academic staffs in PPGEPS from time and frequency perspective is shown. This summary is based on the contents of the column “No.”, “Who” and “When”, from which we can discover two extremes: (1) There is a strong collaboration between the academic staff “F” and the academic staff “A”, who always carry out scientific research activities together; (2) There is lack of collaboration between the academic staff “E” (or “I”) and the others in this academic faculty.

Figure 3
Academic relationships among the academic staffs in PPGEPS from the time and frequency perspective.

As can be seen from Figure 4, the summary of the academic relationships among the academic staffs in PPGEPS from domain and frequency perspective is shown. Considering the research domain of this group, a domain framework that describes the life cycle of a product (Institute of Electrical and Electronics Engineers, 2008Institute of Electrical and Electronics Engineers. (2008). IEEE Std 15288-2008: ISO/IEC/IEEE systems and software engineering: system life cycle processes. Piscataway: IEEE. http://dx.doi.org/10.1109/IEEESTD.2008.4475828
http://dx.doi.org/10.1109/IEEESTD.2008.4...
) is employed as its background. This summary is based on the contents of the column “No.”, “Who” and “Where”, from which we can find one issue: Most of the academic staffs are only contributing to the Concept stage, the Development stage and the Production stage. There is a lack of contributions to the Utilization/Support stage and the Retirement stage. Furthermore, we also can discover that the academic staff “D” and the academic staff “J” have most of their publications in the Development stage and the academic staff “A” and the academic staff “F” show many scientific research activities in the Production stage.

Figure 4
Academic relationships among the academic staffs in PPGEPS from the domain and frequency perspective.

### 3.3.2 The summaries of the research expertise

As can be seen from Figure 5, the summary of the research expertise in PPGEPS from the research object perspective is shown. This summary is based on the contents of the column “No.”, “Who” and “What”, from which we can discover the different research objects that those academic staffs worked on. For example, the academic staffs “G” and “H” were mainly working on the Process Model, which accounts for 38% and 42% of their total research objects respectively.

Figure 5
Research expertise of the academic staffs in PPGEPS from the research object perspective.

As can be seen from Figure 6, the summary of the research expertise in PPGEPS from the research challenge perspective is shown. This summary is based on the contents of the column “No.”, “Who” and “Why”, from which we can find out the different research challenges that those academic staffs intended to address. For example, the academic staff “D” was mainly working on the research challenge that lies in the subject of Product Modeling (28%) and the subject of Sustainability (28%).

Figure 6
The research expertise of the academic staffs in PPGEPS from the research challenge perspective.

As can be seen from Figure 7, the summary of the research expertise in PPGEPS from the research method perspective is shown. This summary is based on the contents of the column “No.”, “Who” and “How”, from which we can discover the different research methods or technologies that those academic staffs can handle. For example, the academic staff “B” utilized many different kinds of algorithms in his publications, in which, the method that related to the Evolutionary Algorithm (21%) is the most used.

Figure 7
The research expertise of the academic staffs in PPGEPS from the research method perspective.

After the discussion and conformation with the research manager and several academic staffs in PPGEPS, these results not only match to the reality in a great measure but also provide evidence-based summaries rather than the experimentalism. Moreover, based on these five summaries, the research manager gave two suggestions to improve the research capability in this academic faculty: (1) advising the academic staff “E” and the academic staff “I” to discover some potential collaboration possibilities with others in this faculty. For example, according to the summaries of their research expertise from the research method and challenge perspectives, the manager suggested the academic staff “E” to find the potential collaboration possibility with the academic staff “B”; (2) advising the academic staff “H” and the academic staff “M” to explore their new research activities in the Support stage and the Retirement stage of a product, according to the summaries of their research expertise from the research object and challenge perspectives.

# 4 Conclusion and future works

To our knowledge, this method is capable of providing reliable and evidence-based summaries for the development of academic faculty rather than the experimentalism. Moreover, some future works are worth noting from the completeness aspect. It includes: (1) To employ or develop a domain ontology that can be used to distinguish and classify the existing or extracted keywords. For example, a knowledge base to deal with the semantic synonymous and subsumable keyword issues. (2) To propose more summarization combinations (other than the existing five kinds) for more precise investigation of scientific research activities. For example, the combination of “No.”, “Who”, “When” and “Why” can be used to show the trends of research challenges that those academic staffs intended to address in several years. (3) To analyze feedbacks from students and professors about the new proposed courses.

# Acknowledgements

The authors would like to thank the financial supports provided by the CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) and the PUCPR (Pontifícia Universidade Católica do Paraná) in Brazil.

• How to cite this article: Loures, E. R., Liao, Y., & Canciglieri Junior, O. (2017). A classification and summarization method for analysis of research activities in an academic faculty. Production, 27(spe), e20162163. http://dx.doi.org/10.1590/0103-6513.216316

# References

• Adler, N., Elmquist, M., & Norrgren, F. (2009). The challenge of managing boundary-spanning research activities: experiences from the Swedish context. Research Policy, 38(7), 1136-1149. http://dx.doi.org/10.1016/j.respol.2009.05.001
» http://dx.doi.org/10.1016/j.respol.2009.05.001
• Arksey, H., & O’Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. http://dx.doi.org/10.1080/1364557032000119616
» http://dx.doi.org/10.1080/1364557032000119616
• Arundel, A., & Constantelou, A. (2006). Conventional and experimental indicators of knowledge flows. In Y. Caloghirou, A. Constantelou & N. Vonortas (Eds.), Knowledge flows in European industry (pp. 45-66). London: Routledge. Retrieved in 21 July 2016, from http://ecite.utas.edu.au/75683
» http://ecite.utas.edu.au/75683
• Bercovitz, J., & Feldman, M. (2011). The mechanisms of collaboration in inventive teams: composition, social networks, and geography. Research Policy, 40(1), 81-93. http://dx.doi.org/10.1016/j.respol.2010.09.008
» http://dx.doi.org/10.1016/j.respol.2010.09.008
• Chen, N.-S., Wei, C.-W., & Chen, H.-J. (2008). Mining e-Learning domain concept map from academic articles. Comput. Educ., 50(3), 1009-1021. http://dx.doi.org/10.1016/j.compedu.2006.10.001
» http://dx.doi.org/10.1016/j.compedu.2006.10.001
• Feldman, M. P., Lanahan, L., & Lendel, I. V. (2014). Experiments in the laboratories of democracy: state scientific capacity building. Economic Development Quarterly, 28(2), 107-131. http://dx.doi.org/10.1177/0891242413490018
» http://dx.doi.org/10.1177/0891242413490018
• Geuna, A., & Martin, B. R. (2003). University research evaluation and funding: an international comparison. Minerva, 41(4), 277-304. http://dx.doi.org/10.1023/B:MINE.0000005155.70870.bd
» http://dx.doi.org/10.1023/B:MINE.0000005155.70870.bd
• He, Z.-L., Geng, X.-S., & Campbell-Hunt, C. (2009). Research collaboration and research output: a longitudinal study of 65 biomedical scientists in a New Zealand university. Research Policy, 38(2), 306-317. http://dx.doi.org/10.1016/j.respol.2008.11.011
» http://dx.doi.org/10.1016/j.respol.2008.11.011
• Higher Education Funding Council for England. (2009). Research Excellence framework: second consultation on the assessment and funding of research. Bristol: HEFCE. Retrieved in 21 July 2016, from http://www.hefce.ac.uk/pubs/year/2009/200938/
» http://www.hefce.ac.uk/pubs/year/2009/200938/
• Hoekman, J., Frenken, K., & Tijssen, R. J. W. (2010). Research collaboration at a distance: changing spatial patterns of scientific collaboration within Europe. Research Policy, 39(5), 662-673. http://dx.doi.org/10.1016/j.respol.2010.01.012
» http://dx.doi.org/10.1016/j.respol.2010.01.012
• Hulth, A. (2003). Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (Vol. 10, pp. 216-223). Stroudsburg, USA. http://dx.doi.org/10.3115/1119355.1119383
» http://dx.doi.org/10.3115/1119355.1119383
• Institute of Electrical and Electronics Engineers. (2008). IEEE Std 15288-2008: ISO/IEC/IEEE systems and software engineering: system life cycle processes. Piscataway: IEEE. http://dx.doi.org/10.1109/IEEESTD.2008.4475828
» http://dx.doi.org/10.1109/IEEESTD.2008.4475828
• Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 1(1), 10. PMid:22587960. http://dx.doi.org/10.1186/2046-4053-1-10
» http://dx.doi.org/10.1186/2046-4053-1-10
• King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of Information Science, 13(5), 261-276. http://dx.doi.org/10.1177/016555158701300501
» http://dx.doi.org/10.1177/016555158701300501
• Li, F., Miao, Y., & Yang, C. (2014). How do alumni faculty behave in research collaboration? An analysis of Chang Jiang Scholars in China. Research Policy, 44(2), 438-450. http://dx.doi.org/10.1016/j.respol.2014.09.002
» http://dx.doi.org/10.1016/j.respol.2014.09.002
• Liao, Y., Lezoche, M., Loures, E. R., Panetto, H., & Boudjlida, N. (2014). Formal semantic annotations for models interoperability in a PLM environment. In Proceedings of the 19th World Congress of the International Federation of Automatic Control (IFAC 2014) (pp. 2382-2393), Cape Town, South Africa. http://dx.doi.org/10.3182/20140824-6-ZA-1003.02551
» http://dx.doi.org/10.3182/20140824-6-ZA-1003.02551
• Matsuo, Y., & Ishizuka, M. (2004). Keyword extraction from a single document using word co-occurrence statistical information. International Journal of Artificial Intelligence Tools, 13(1), 157-169. http://dx.doi.org/10.1142/S0218213004001466
» http://dx.doi.org/10.1142/S0218213004001466
• Mullen, B. (1986). Basic meta-analysis: description of a statistical package. Behavior Research Methods, Instruments, & Computers, 18(2), 165-167. http://dx.doi.org/10.3758/BF03201018
» http://dx.doi.org/10.3758/BF03201018
• Smeby, J. C., & Try, S. (2005). Departmental contexts and faculty research activity in Norway. Research in Higher Education, 46(6), 593-619. http://dx.doi.org/10.1007/s11162-004-4136-2
» http://dx.doi.org/10.1007/s11162-004-4136-2
• Smith, S., Ward, V., & House, A. (2011). Impact’ in the proposals for the UK’s research excellence framework: shifting the boundaries of academic autonomy. Research Policy, 40(10), 1369-1379. http://dx.doi.org/10.1016/j.respol.2011.05.026
» http://dx.doi.org/10.1016/j.respol.2011.05.026
• Yih, W., Goodman, J., & Carvalho, V. R. (2006). Finding advertising keywords on web pages. In Proceedings of the 15th International Conference on World Wide Web (WWW ’06) (pp. 213-222), Edinburgh, Scotland. http://dx.doi.org/10.1145/1135777.1135813
» http://dx.doi.org/10.1145/1135777.1135813

# Publication Dates

• Publication in this collection
2017