Meeting the assumptions of statistical tests: an important and often forgotten step to reporting valid results

PRACTICAL SCENARIO

A secondary analysis of a randomized controlled trial was conducted to evaluate whether atopic status was associated with asthma severity and asthma control among inner-city adolescents in the USA.¹ To answer this question, the authors evaluated the differences between atopic and non-atopic patients, in terms of asthma control and asthma severity scores, using a t-test, and reported the results as means ± SDs. The results showed that the atopic patients, when compared with non-atopic patients, had similar asthma control scores (18.1 ± 4.2 vs. 18.2 ± 3.7; p = 0.95) but worse asthma severity scores (5.5 ± 2.9 vs. 4.7 ± 2.8; p = 0.04).

BACKGROUND

As part of the process of answering research questions using quantitative methods, investigators select a statistical analytical approach based on various characteristics of the study, such as the nature of the variables collected (e.g., continuous, categorical, time-to-event variables), as well as the study design. Once the analysis is completed, it is expected that investigators take an additional step in the analysis process to make sure that the a priori assumptions of the statistical test selected are met in the dataset assembled for the study.

All statistical tests have underlying assumptions that need to be met so that the test provides results that are valid (without unacceptable error) regarding the parameter the test is calculating (e.g., mean, proportion, odds ratio, etc.). In our example, the authors used a t-test to calculate the mean and the standard deviation of asthma control and severity asthma scores in atopic and non-atopic patients using data collected from the study population as a means to represent the truth in similar patients from the source population (adolescents with asthma in the USA). This process, called inference, is only valid if the assumptions of the statistical test are met (Table 1).

Table 1
Example of assumptions of a statistical test.

Statistical test	Assumption	How to corroborate
t-Test	Sampling: The participants in the study are randomly sampled from the source population.	Check the protocol.
	Sample size: The sample size calculated for the study is achieved.	Check sample size calculation in the protocol and check if sample size was reached by the number of participants included in the study.
	Normal distribution: The scale of measurement of the outcome variable is continuous and is normally distributed (or at least symmetric).	Conduct descriptive statistics on the outcome variable and create a graph showing the distribution which should follow a bell curve.
	Homogeneity of variances: The variance (standard deviation) of the data collected on the continuous variable across the two comparison groups is similar.	Use valid statistical methods to test for homogeneity.

It is good practice, as investigators, to acknowledge that the assumptions of the statistical tests used to answer their research question have been evaluated and whether they were met or not. If the assumptions of the tests are met, which should be reported in the results section of the study, this assures the scientific community that the results of the study have met one of the important criteria related to their validity. However, it has been suggested that the assumptions of statistical techniques are often not checked² or reported. Reasons for not assessing assumptions include: 1. researchers being unaware of the assumptions of the statistical tests used in the study, such as a t-test, ANOVA, or regression analysis; 2. researchers being unaware of standard approaches used to check assumptions of statistical tests and evaluate if they are violated or not; 3. researchers being unaware of how to remedy violations of the assumptions of a statistical model or how to select a new test when violations cannot be remedied; and 4. researchers being confident in the robustness of the statistical test used and choosing not to check its assumptions.

As educators and investigators, we all need to contribute to the overall goal of reporting high quality research conducted among the populations we serve. Testing the assumptions of statistical tests or models used to answer our research questions is a good start!

REFERENCES

• ¹
  Arroyave WD, Rabito FA, Carlson JC, Sever ML, Lefante J. Asthma severity, not asthma control, is worse in atopic compared with nonatopic adolescents with asthma. Ann Allergy Asthma Immunol 2016;116(1):18-25. https://doi.org/10.1016/j.anai.2015.10.015
  » https://doi.org/10.1016/j.anai.2015.10.015
• ²
  Hoekstra R, Kiers HA, Johnson A. Are assumptions of well-known statistical techniques checked, and why (not)? Front Psychol. 2012;3:137. https://doi.org/10.3389/fpsyg.2012.00137
  » https://doi.org/10.3389/fpsyg.2012.00137

Publication Dates

• Publication in this collection
  Sep-Oct 2018