Crisis in neuroimaging: is neuroimaging failing 15 years after the decade of the brain?

In 1989, the United States Congress declared that the 1990s were to be designated the decade of the brain. Indeed, great advances were made in that decade with the emergence of new methods in neuroimaging research. These more reliable and accessible techniques for using in vivo brain images in neuroscience led to new discoveries on the neural mechanisms of psychiatric disorders. The extent to which neuroimaging contributed to a better understanding of mental disorders can be exemplified by how it influenced our understanding of, and research perspectives in, schizophrenia. Early work on brain alterations during the course of schizophrenia lead to the neurodevelopmental hypothesis of this condition,¹ a notion expanded further during the 1990s with magnetic resonance imaging (MRI) studies in first-episode patients, who were found to already show differences compared to healthy controls.² This substantiated the need to conduct research in even earlier stages, before the onset of full-blown psychotic disorders. Since then, it has been shown that patients in at-risk groups already show differences in brain morphology and functioning when compared to healthy controls.³ However, there is one major question that continues to challenge clinicians and researchers in at-risk approaches. The conversion rate to first episode is low, with approximately one-third of at-risk patients transitioning to a psychosis syndrome after 3 years of follow-up.³ These relatively low conversion rates make planning interventions in at-risk groups particularly difficult. Ongoing studies aim to develop tools to use neuroimaging to help stratify the chance of at-risk patients progressing to first-episode psychosis.³

With the availability of free and more user-friendly imaging processing softwares and the cost reduction of MRI, neuroimaging research became a “hot topic,” and the number of articles published in the field increased hugely from 2000 onward (Figure 1).

Figure 1
Graphic representation of results per year in the PubMed database using the keywords psychiatry and neuroimaging. An exponential increase is noted after the release of user-friendly research-oriented software in the early 2000s.

However, recent research has called into question a significant portion of published functional MRI (fMRI) findings. A recent paper⁴ claimed that approximately 40,000 neuroimaging studies using fMRI results might be wrong or biased. This strong statement attracted great attention in the general media and social networks, which led to the rushed conclusion that most neuroimaging reports are wrong and that this was caused by a software bug. What the authors of this study actually showed is how easy it is to obtain false-positive clusters when using parametric methods of correction for multiple comparisons in neuroimaging, depending on the threshold used for analysis. There was in fact a bug reported in a popular neuroimaging software (AFNI – 3dclustsim), but this has already been corrected and the high false-positive rate reported was not due to the bug alone. The problem of multiple testing is known to neuroimaging researchers and addressed by most neuroimaging softwares. This is especially true in whole-brain analysis, an approach that consists of performing statistical tests across each small piece of data contained in a brain image. The problem is that statistical tests are not all independent: rather, they are circumscribed to an anatomical space (i.e., one data point is more likely to covariate with the neighboring data point than with a distant data point in another brain region). Taking this into account, cluster-based approaches have been developed. The idea behind such approaches is that, if many neighboring data points form a positive cluster, this is unlikely to have occurred by chance. Although the underlying concept remains valid, most tools used to perform this form of correction have been validated using simulated data, which may behave differently from actual data. In fact, the article by Eklund et al.⁴ was not the first to show this; other research has demonstrated that some neuroimaging methods can lead to high false-positive rates.⁵ It is important to note that the findings of Eklund et al.⁴ did not go unquestioned. In fact, they have sparked a heated debate regarding known methodological issues that could lead to inflated false-positive rates, and that significance levels should not be interpreted without paying attention to effect sizes⁶ and methodological parameters used in obtaining neuroimaging results. Finally, an erratum submitted by the authors removed their original claim that 40,000 results could be wrong; however, this was not so broadly reported to the general public.

The fact that the statistics used in most neuroimaging reports are not error-free is well known to investigators. One of the first works to point out statistical caveats in the field was the “voodoo correlation” report by Vul et al.,⁷ which described how double-dipping can lead to spurious results. Double-dipping is the practice of conducting a second statistical analysis based on the results of a first significant analysis, thus biasing the result of the second analysis.⁷ For example, it is common practice to conduct a whole-brain analysis, extract the single voxel of the whole brain which yielded the smallest p-value on statistical testing, and, using measures obtained from this single voxel, perform further statistical testing to prove the strength of the finding being described.

This leads to yet another statistics-related problem in neuroimaging. Most neuroimaging data are used to test multiple hypotheses, but this is seldom taken into account. One can, for example, test whether a symptom correlates with brain measures in one analysis, carry out a different analysis with another brain symptom, and treat both analyses as completely independent. This problem can be especially hard to tackle because such findings may be reported in different papers.

To the psychiatrist, the more technical discussions of neuroimaging can be difficult to follow. What should the clinician do with this information? Should all the neuroimaging findings of the past two decades be dismissed as false positives? And, more importantly, has neuroimaging research failed to deliver consistent results and gone back to square one? It is the authors’ view that the answer to both of these questions is no. As in many other fields of science, neuroimaging may be affected by low replicability and false-positive findings. However, we believe that there is a large body of evidence for brain alterations linked to psychiatric disorders, which have been replicated in more than one study and using different methods of analysis, and that these findings should be trusted. The psychiatric and behavioral applications of neuroimaging remain an exciting field, still under construction after more than two decades of intensive research. Methods are evolving, and new and more informative approaches are being developed. Efforts are being made to understand how different regions of the brain interact with each other (e.g., graph analysis and dynamic causal modeling) and how information from different modalities can be integrated, as well as to develop statistical tools capable of taking into account the issues raised by recent work. There are also initiatives to standardize and create best-practice guidelines for conducting studies and reporting neuroimaging findings.⁸ These actions should lead to even greater progress toward understanding the biological basis of mental disorders. As noted earlier in this editorial, neuroimaging was a powerful tool to help investigate hypothesis on mental disorders (e.g., the neurodevelopmental hypothesis of schizophrenia) and paved the way to new research strategies (e.g., investigation of at-risk groups). We hope that, in the not so distant future, it may be used to help clinicians deliver better, personalized care in psychiatry.

Acknowledgements

The authors are receive support from Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP; JRS – grants 2013/10498-6 and 2013/00506-1; APJ – grant 2013/08531-5) and to Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; APJ – grants 442026/2014-5 and 312984/2014-6).

References

• ¹
  Murray RM, Lewis SW. Is schizophrenia a neurodevelopmental disorder? Br Med J (Clin Res Ed). 1987;295:681-2.
• ²
  Ellison-Wright I, Glahn DC, Laird AR, Thelen SM, Bullmore E. The anatomy of first-episode and chronic schizophrenia: an anatomical likelihood estimation meta-analysis. Am J Psychiatry.2008;165:1015-23.
• ³
  McGuire P, Sato JR, Mechelli A, Jackowski A, Bressan RA, Zugman A. Can neuroimaging be used to predict the onset of psychosis? Lancet Psychiatry. 2015;2:1117-22.
• ⁴
  Eklund A, Nichols TE, Knutsson H. Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates. Proc Natl Acad Sci U S A. 2016;113:7900-5.
• ⁵
  Meyer-Lindenberg A, Nicodemus KK, Egan MF, Callicott JH, Mattay V, Weinberger DR. False positives in imaging genetics. Neuroimage. 2008;40:655-61.
• ⁶
  Chen G, Taylor PA, Cox RW. Is the statistic value all we should care about in neuroimaging? [Internet]. bioRxiv. 2016 Jul 15. http://dx.doi.org/10.1101/064212
  » http://dx.doi.org/10.1101/064212
• ⁷
  Vul E, Harris C, Winkielman P, Pashler H. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect Psychol Sci. 2009;4:274-90.
• ⁸
  Nichols TE, Das S, Eickhoff SB, Evans AC, Glatard T, Hanke M, et al. Best practices in data analysis and sharing in neuroimaging using MRI [Internet]. bioRxiv. 2016 May 20. http://dx.doi.org/10.1101/054262
  » http://dx.doi.org/10.1101/054262

Publication Dates

• Publication in this collection
  Dec 2016