1.

# Introduction

The purpose of the research is to create new knowledge. But we do not want just any new knowledge; it must also be useful and have an impact. In academia, we typically measure impact through how often the new knowledge is transmitted and especially by how many times it is cited by others or used productively. Most researchers are on the constant lookout for new ideas that will attract attention and have an impact. But it is not clear where good ideas come from. When we try to remember how we came about our most creative and impactful ideas in the past, it is often not obvious, even to ourselves, how we did it. And there is evidence that good ideas are getting harder to find. Bloom, Jones, Reenen, and Webb (2017)Bloom, N., Jones, C.I., Reenen, J.V., & Webb, M. (2017). Are ideas getting harder to find? NBER Working Paper No. 23782, September. show that in a wide range of economic sectors (agricultural, medical and technological), ever more research effort has historically been necessary to generate less and less innovation, thus denoting a sharp decline in research productivity. Wuchty, Jones, and Uzzi (2007)Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science. 316, 1036–1038. use data from over 19 million publications over five decades to show that the process of knowledge creation has fundamentally changed, with most high-impact research now produced by teams instead of individuals. Even Nobel Prizes are increasingly awarded to more than one person at a time, with the average age of when they made their greatest discoveries having significantly increased in recent decades (Jones & Weinberg, 2011Jones, B.F. and Weinberg, B.A. (2011), Age dynamics in scientific creativity, PNAS, 108, 18910–18914.).

Given the increasing difficulty of generating impact, it would be useful if there were reliable guidelines or strategies for how to produce new ideas and innovations. A plethora of books, videos, lectures, specialists, programs, courses and initiatives exist across all fields and sectors that try to answer that question, not all of them of the best quality. In this article, I look at research that has effectively contributed to our understanding of where good ideas come from by examining hard data from specific creativity-intensive sectors such as science, academia, music and patents. Although this research does not uncover the secret formula for creative success, it does point to a robust pattern that provides some insights as to how innovation and impact are created. Understanding this pattern can be useful for individual researchers, organizations and for public policy.

# Innovation through recombination

A common view of where innovation comes from is the lone genius pulling brilliant ideas out of thin air. But much research has shown that the production of new knowledge is inherently a process of recombination of existing knowledge, be that in technology and science (Arthur, 2007Arthur, W.B. (2007). The structure of invention. Research Policy, 36, 274–287.; Mokyr, 2018Mokyr, J. (2018) A Culture of Growth: The Origins of the Modern Economy. Princeton, NJ: Princeton University Press.), economies (Hidalgo & Hausmann, 2009Hidalgo, C., & Hausmann, R. (2009), The building blocks of economic complexity, Proceedings of the National Academy of Sciences of the United States, 106, 10570–10575.; Romer, 1994Romer, P.M. (1994). The origins of endogenous growth. The Journal of Economic Perspectives, 8, 3–22.; Weitzman, 1998Weitzman, M.L. (1998). Recombinant growth. Quarterly Journal of Economics, 113, 331–360.), academia/research (Uzzi, Mukherjee, Stringer, & Jones, 2013Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342, 468–472.; Wang, Veugelers, & Stephan, 2017Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46, 1416–1436.), music (Askin & Mauskapf, 2017Askin, N., & Mauskapf, M. (2017). What makes popular culture popular? Product features and optimal differentiation in music. American Sociological Review, 82, 910–944.; Mueller, 2018Mueller, B. (2018). Whered you get that idea? Determinants of creativity and impact in popular music. Working Paper, Department of Economics, University of Brasilia, Brasilia, Brazil.) and cuisine (Ahn, Ahnert, Bagrow, & Barabási, 2011Ahn, Y.Y., Ahnert, S.E., Bagrow, J.P., & Barabási, A.L. (2011). Flavor network and the principles of food pairing. Scientific Reports, 1, 196. Available from https://www.nature.com/articles/srep00196
https://www.nature.com/articles/srep0019...
), among several other areas. Recombination is, of course, the central process through which novelty is created in living creatures (Darwin, 1859Darwin, C. (1859), On the origin of species by means of natural selection, or preservation of favoured races in the struggle for life, London, United Kingdom: John Murray.), for although mutations are also necessary for evolution, recombination (or cross-over) is involved in every mating, whereas mutations occur in less than one in a million individuals (Holland, 1995Holland, J.H. (1995). Hidden order: How adaptation builds complexity, Redwood City, CA: Addison-Wesley.). The creation of novelty can thus be conceived as taking place primarily in what Kauffman (1995)Kauffman, S.A. (1995), At home in the universe: the search for laws of self-organization and complexity, New York, NY: Oxford University Press. has called the adjacent possible, that is, the set of all things that can be created given our current knowledge, which Johnson (2010)Johnson, S. (2010), Where good ideas come from: The natural history of innovation, New York, NY: Penguin Group. explains as:

[…] a kind of shadow future, hovering on the edges of the present state of things, a map of all the ways in which the present can reinvent itself […] [the adjacent possible] captures both the limits and the creative potential of change and innovation.

In (Mueller, 2018Mueller, B. (2018). Whered you get that idea? Determinants of creativity and impact in popular music. Working Paper, Department of Economics, University of Brasilia, Brasilia, Brazil.), I provide empirical evidence that in the area of pop music, recombination into the adjacent possible is in fact a major component in the creative process. I used data from a podcast (Song Exploder[1]) where musicians describe in detail how they composed a specific song. By mining these narratives, I was able to identify several processes or elements of the creative process recurrently mentioned by the composers. The most prevalent of these was precisely recombination, with over 71 per cent of the narratives explicitly indicating which existing songs had inspired their own composition. In addition, half of the narratives indicated a key role played by some other person not usually associated with that composer (such as a producer or a studio musician), which is a kind of recombination, not of songs but of brains or perspectives.

The fact that the recombination of existing knowledge is a major means through which novelty is created is interesting but tells us little about which patterns or strategies for combining what we already know are more fruitful and impactful. To get at this issue, I used a map of over 1,800 musical genres produced by Every Noise at Once[2] using data from millions of songs on Spotify.[3] This map allowed me to produce statistics characterizing how far and in which directions composers sampled from their own usual genres. These statistics were then used to test which kinds of recombination were most likely to lead to greater impact as measured by average number of daily YouTube views of the song since its release. The results indicated a clear pattern. The songs’ success and impact were statistically more likely if the inspiration songs were of a different genre, but not too distinct. That is, songs inspired by songs from similar genres were less likely to have great impact. But songs inspired by genres that were too distant similarly failed. It was those that presented novelty but at the same time had familiarity that hit the sweet spot. A bimodality coefficient showed that a particularly good strategy was to sample simultaneously both near and far in the space of all genres. In the terms introduced in the management literature by March (1991)March, J.G. (1991), Exploration and exploitation in organizational learning, Organization Science, 2, 71–87., this result confirms the importance of carefully dosing the mix of exploration versus exploitation in the process of creating novelty.

Similar results have been found for variety of different areas. Uzzi et al. (2013)Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342, 468–472. used data from 15,000 academic journals to analyze which citation patterns were associated with greater impact. They conclude that hit papers have tend to simultaneously exhibit conventionality and novelty, a signature they call “atypical combinations.” Youn, Strumsky, Bettencourt and Lobo (2015Youn, H., Strumsky, D., Bettencourt, L. M. A., & Lobo, J. (2015). Invention as a combinatorial process: Evidence from US patents. Journal of the Royal Society Interface, 12. https://doi.org/10.1098/rsif.2015.0272
https://doi.org/10.1098/rsif.2015.0272...
, p. 1) use patent data from 1790 to 2010 and find “that the combinatorial inventive process exhibits an invariant rate of ‘exploitation’ (refinements of existing combinations of technologies) and ‘exploration’ (the development of new technological combinations).” Askin and Mauskapf (2017)Askin, N., & Mauskapf, M. (2017). What makes popular culture popular? Product features and optimal differentiation in music. American Sociological Review, 82, 910–944. also analyze popular music but through a different method that classifies songs according to a set of ten features, which it uses to produce a typicality index. This index is then the key variable to explain songs’ chart success. They find that hit songs are those that best manage the trade-off between being recognizable and being different, which they call “optimal differentiation.” Barron et al. (2018) use machine learning to uncover the process through which new ideas emerged in the aftermath of the French Revolution. They analyze over 40,000 speeches during the Revolution’s first parliament, in which highly influential political ideas were emerging, i.e. liberté, égalité and fraternité. They create an index of the novelty of ideas and an index of transience, that is, how long new ideas endure. They show that the more novel an idea, the more quickly tended to fade. However, ideas that resonated and had impact tended to have higher novelty. Thus, the conclusion is that the impact typically requires a measure of novelty, but novelty on its own is usually not enough.

# Personal and policy implications

If one accepts the conclusion that good ideas disproportionately follow a signature that combines both novelty and conventionality, what recommendations emerge for individual researchers, for organizations and for public policy? At a personal and organizational level, these results suggest seeking a balance between exploitation (researching in your main area/topic and using your usual set of methods) and exploration (venturing to new subjects and methods). Arguably, most people bias this balance toward exploitation, so the course correction requires reaching out to unfamiliar territory, yet without becoming unrecognizable to those in your own area. This inbreeding bias is often because of our parochial nature to distrust that which is unfamiliar. But it is also often built into policy, incentives and regulation. The ranking of journals used by funding agencies and universities for hiring and promotion decisions (such as the Qualis list in Brazil[4 4. The CAPES Qualis system aims to classify scientific production done by Brazilian researchers in all fields of knowledge. All the scientific journals are listed according to the criteria A1 (the highest), A2, B1, B2, B3, B4, B5 or C (not considered as valid in the list). In the Management area, currently there are no Brazilian journals listed as A1. The list can be seen at https://sucupira.capes.gov.br/sucupira/public/index.xhtml ]) seems like a sensible way to reward effort and better research. But often this practice partitions researchers into silos and has the effect of curtailing the experimentation and recombination that the research above has shown to be a major determinant of creativity and impact.

In Bias against novelty in science: A cautionary tale for users of bibliometric indicators, Wang et al. (2017)Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46, 1416–1436. measure the novelty of all articles published in the Web of Science in 2001 in all disciplines and compare that to a measure impact (citations). They show that novel research is riskier (often leads to little impact) but is more likely on average to lead to a top 1 per cent highly cited paper. Interestingly though, this recognition takes more time to be realized than more conventional research, and it is disproportionately published in low impact factor journals as well as being published in ‘foreign fields’ instead of the researchers’ “home field.” This implies that narrow-minded grant and hiring decisions based on what are supposedly meritocratic parameters, as well as insular peer-review processes, may be effectively dampening the incentives for the type of research that has a greater probability to expand our adjacent possible.

# Notes

• 1.
• 2.
• 3.
• 4.
The CAPES Qualis system aims to classify scientific production done by Brazilian researchers in all fields of knowledge. All the scientific journals are listed according to the criteria A1 (the highest), A2, B1, B2, B3, B4, B5 or C (not considered as valid in the list). In the Management area, currently there are no Brazilian journals listed as A1. The list can be seen at https://sucupira.capes.gov.br/sucupira/public/index.xhtml

# References

• Ahn, Y.Y., Ahnert, S.E., Bagrow, J.P., & Barabási, A.L. (2011). Flavor network and the principles of food pairing. Scientific Reports, 1, 196. Available from https://www.nature.com/articles/srep00196
» https://www.nature.com/articles/srep00196
• Arthur, W.B. (2007). The structure of invention. Research Policy, 36, 274–287.
• Askin, N., & Mauskapf, M. (2017). What makes popular culture popular? Product features and optimal differentiation in music. American Sociological Review, 82, 910–944.
• Bloom, N., Jones, C.I., Reenen, J.V., & Webb, M. (2017). Are ideas getting harder to find? NBER Working Paper No. 23782, September.
• Darwin, C. (1859), On the origin of species by means of natural selection, or preservation of favoured races in the struggle for life, London, United Kingdom: John Murray.
• Hidalgo, C., & Hausmann, R. (2009), The building blocks of economic complexity, Proceedings of the National Academy of Sciences of the United States, 106, 10570–10575.
• Holland, J.H. (1995). Hidden order: How adaptation builds complexity, Redwood City, CA: Addison-Wesley.
• Johnson, S. (2010), Where good ideas come from: The natural history of innovation, New York, NY: Penguin Group.
• Jones, B.F. and Weinberg, B.A. (2011), Age dynamics in scientific creativity, PNAS, 108, 18910–18914.
• Kauffman, S.A. (1995), At home in the universe: the search for laws of self-organization and complexity, New York, NY: Oxford University Press.
• March, J.G. (1991), Exploration and exploitation in organizational learning, Organization Science, 2, 71–87.
• Mokyr, J. (2018) A Culture of Growth: The Origins of the Modern Economy Princeton, NJ: Princeton University Press.
• Mueller, B. (2018). Where`d you get that idea? Determinants of creativity and impact in popular music. Working Paper, Department of Economics, University of Brasilia, Brasilia, Brazil.
• Romer, P.M. (1994). The origins of endogenous growth. The Journal of Economic Perspectives, 8, 3–22.
• Smith, E. (2010), The adjacent possible: Practically efficient blog Available from https://www.edge.org/conversation/stuart_a_kauffman-the-adjacent-possible
• Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Atypical combinations and scientific impact. Science, 342, 468–472.
• Wang, J., Veugelers, R., & Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46, 1416–1436.
• Weitzman, M.L. (1998). Recombinant growth. Quarterly Journal of Economics, 113, 331–360.
• Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science. 316, 1036–1038.
• Youn, H., Strumsky, D., Bettencourt, L. M. A., & Lobo, J. (2015). Invention as a combinatorial process: Evidence from US patents. Journal of the Royal Society Interface, 12 https://doi.org/10.1098/rsif.2015.0272
» https://doi.org/10.1098/rsif.2015.0272

# Publication Dates

• Publication in this collection
15 July 2019
• Date of issue
Apr-Jun 2019