Therapeutic impact means: relative versus absolute

Coining the term “statistical significance” was a major historical mistake. I don’t know who had this terrible idea.

Significance comes from the Latin significans, which connotes “value,” “importance.” When we statistically test an association, we do not evaluate the importance of the association, only its veracity. Nor can we claim that a “significant” association is causal, much less the relevance of that causality. Thus, the statistical significance only suggests that it would be very unusual that such an association showed up by chance if the null hypothesis were true.

To make things even more complicated, the human mind works in a more categorical way (yes or no) than quantitative. For example, we are happy with the observation “this treatment reduces mortality,” and very few times we seek to know how much. Therefore, I propose that the term “statistical significance” is urgently replaced by “statistical validity.”

To achieve statistical validity, we show that a phenomenon is true. The second step, then, would be to evaluate the real significance (relevance) of the phenomenon. For this, we need to assess the relative and absolute effect. In this article, we will review the meaning and importance of these measures.

THE SIZE OF THE EFFECT

The human mind is more affective than quantitative. The Nobel Laureate psychologist Daniel Kahneman described the affect bias, which is one of the causes of mistaking risk for injury, we overestimate small risks and underestimate high risks. Aspects related to emotion (affect) interfere in our perception of the reality, as Immanuel Kant said.

We are more afraid of getting on a plane than entering the bathroom to take a shower, despite the risk of death from a fall in the bathroom be far greater than the risk of death by plane crash. We fear more yellow fever than the flu, even though the risk of death from influenza is much greater than death from yellow fever.

When describing a treatment, we usually do not quantify the intrinsic benefit, we just characterize it. We simply say, “this treatment is beneficial,” or “this treatment reduces mortality.” Yes, but how much does it reduce?

When we quantify, we run the risk of overvaluing moderate-impact treatments or underestimate high-impact treatments. And that comes attached to the “affective” and not the quantitative way we analyze our behaviors.

HOW TO MEASURE THE SIZE OF THE EFFECT?

The traditional evidence-based medicine approach emphasizes the absolute reduction of the risk and the number needed to treat (NNT) as the main measures of the size of the effect, to the detriment of the relative reduction of the risk and of the relative risk.

It is common to say, “the relative misleads, what counts is the absolute”. I often use the inheritance example. If I have won 50% of the fortune of an uncle (relative), can I say that I became rich? It seems a lot, but if the fortune is 1 real, I have won only 50 cents. What counts is the absolute.

But that’s only part of the story. The relative is of great importance and essential for medical thinking. In fact, the absolute risk reduction (with which we calculated the NNT - Number Needed to Treat) is not an intrinsic property of the treatment, it is a property of the patient receiving the treatment. For one single treatment, the NNT varies from patient to patient, depending on the baseline risk. Though, we can say that the treatment does not have NNT. Who has NNT is that type of patient who will receive that treatment.

In fact, the intrinsic property of the treatment is the relative risk reduction, which tends to be constant in different risk subgroups. Usually, subgroup analysis does not show an interaction between baseline risk and the relative treatment impact.

A small-effect treatment (relative risk reduction) can provide a great absolute reduction (small NNT) if applied to a population of very high risk. Similarly, a big-effect treatment may have a small absolute reduction (ARR) if applied to a low-risk population. Therefore, the relative reduction (RRR) shows the size of the intrinsic effect of the treatment, while the absolute reduction shows the impact of the treatment on a certain type of patient, with a certain kind of baseline risk. We can say that the RRR translates the effect size and the ARR, the impact of the treatment. Therefore, we must measure the size of the intrinsic effect of the treatment by the relative risk reduction, while the NNT is the concrete impact in a given patient.

If we had the chance to know just one information about a treatment, which would we choose? The relative or the absolute reduction?

The relative, of course. Because knowing the relative reduction, one can calculate the absolute reduction of each patient individually, provided that we know the patient’s absolute risk.

For example, let’s say that the relative risk reduction is 33%. Based on a risk score, we estimate 10% as the patient’s baseline risk. Thus, the absolute risk reduction of this patient is 33% x 10%=3.3% (NNT = 100/3.3=30).

As a reference for the analysis, good treatments show a relative risk reduction of around 30 to 40%.

ANGIOTENSIN-CONVERTING ENZYME INHIBITOR IN HEART FAILURE (ACEI)

It is surprising to note that the ACEI inhibitor treatment in heart failure is of small effect size. According to the clinical trial SOLVD¹,{Yusuf, 1991 # 125} the relative risk reduction of ACEI is only 16% lower than most cardiological treatments that work. Almost nobody pays attention to it, because the mortality of the disease is high, giving a good NNT. This is a small-effect treatment, but of reasonable impact due to the severity of the disease.

If I say that the absolute reduction in SOLVD was 4.5%, it looks pretty good. However, the picture changes quite a lot if, instead of showing the absolute reduction, we show the numbers of each group. In the placebo group, the mortality rate was 39.7%, and this reduced to 35.2%. Notice that these two numbers are not so different. Many people without enalapril die (slightly more than 1/3 of the patients), but many people continue to die with enalapril (slightly more than 1/3 of the patients). It does not change much. When we look from this perspective, we see that the size of the treatment effect is small.

Moreover, the confidence interval of the relative risk reduction presented in that study of moderate size (2,500 patients) is wide, from 5 to 20%. Therefore, this treatment can provide a relative risk reduction as low as 5%. Yet, the extreme upper confidence interval (20%) is not so different than the spot measure of 16%. As far as the size of the effect, it is important to look at the accuracy of the estimate described by the confidence interval.

I am not downgrading the importance of this treatment in heart failure, even because ACEI also helps to control the symptoms. But it’s important to have the perspective of the size of the effect, alongside the perspective of NNT.

This perspective reduces the affect bias in favor of the ACE inhibitor, helping us to get our “feet on the ground” and allowing better analysis of the risk-benefit trade-off. We will be thriftier when facing certain patients, such as hypotensive (susceptible to syncope) or with a certain degree of renal dysfunction. With no anguish, we will insist less on high doses (which could trigger syncope episodes) when we think of the panacea of a treatment.

It is very interesting to revisit these data from the past. In the cardiology mind, ACEI is a panacea. It was 1988. I was in the second year of the school of medicine when the CONSENSUS study was published in the New England Journal of Medicine², a seminal clinical trial to test of this hypothesis, always cited to support the ACEI effectiveness in congestive heart failure (CHF). But in fact, this is a tiny study (only 253 patients), which was prematurely interrupted with only 118 outcomes (truncated with less than 200 outcomes is a risk of inaccuracy). That study showed a relative risk reduction of 40%. That’s what remained printed in the affective mind of the cardiologists.

One of the best ways to retain learning is to make it happen with emotion. A child burned by touching a hot pot (trauma) will certainly learn that a pot may burn the hand. When CONSENSUS was published, the news about the relative risk reduction of 40% came as such a novelty that thrilled, we retained this information that will be forever ingrained in our minds. Then, SOLVD came, a larger study that showed a more accurate value of 16% relative risk reduction. But it wasn’t the 16% (small effect) that remained in our affective memory, but the 40% that overwhelmed more and overwhelmed first.

NOVEL ORAL ANTICOAGULANTS

There is a common misconception about these drugs. We believe that their biggest advantage is the convenience of not requiring the prothrombin time tests. Usually, we think that the effectiveness of the novel oral anticoagulants is equivalent to the traditional and cheap warfarin. Therefore, we present the two options to the patient: a convenient and high-cost drug versus another less convenient and of low cost.

But this common reasoning rules out an important fact. The biggest advantage of these drugs is not convenience. The biggest advantage is its superior effect compared to warfarin. In reality, these drugs, when used in its optimal dose, are much better than warfarin. This superiority is more important than the so-mentioned convenience.

In fact, it is very difficult to show the superiority of a new treatment in relation to a traditional and effective treatment. And if the new treatment is better than the traditional, this superiority tends to be small. Unlike the usual, the novel anticoagulants are much better than warfarin. In the RELY study³, the dose of 150 mg of dabigatran promoted a relative risk reduction of embolic events of 34% in patients with atrial fibrillation, something that is at the same level of good treatment compared to placebo. This is almost unprecedented in the treatment versus treatment comparison. Likewise, the ARISTOTLE study⁴ shows that apixaban promotes a relative reduction of 21% when compared to warfarin.

What I mean is that using warfarin instead of a novel anticoagulant is the same as choosing a worse treatment. So, putting convenience versus price as the main trade-off to this kind of shared decision is a mistake. The right trade-off is efficacy versus price, and this efficacy comes with more convenience.

The main focus on convenience, to the detriment of obvious superiority, is an example of mistake due to the lack of perspective of the relative risk reduction.

THE PATIENT’S VIEW

This article has reviewed the main concepts about impact measures. Finally, we should consider that the most relevant measure is more subjective and difficult to quantify. The most relevant measure of the significance of an approach is the patient’s happiness with the final outcome. There’s no point in showing a risk reduction if the patient is free from the outcome, but imprisoned in his/her dissatisfaction.

Relative and absolute risk analysis does not provide all the answers. First, they are probabilistic, not a guarantee. Second, the prevention of the undesired outcome can be mediated by an approach that brings discomfort and patient’s dissatisfaction if it is against his/her values.

In the end, the most important is that our recommendations are primarily influenced by the patient’s preference.

REFERENCES

• ¹
  Yusuf S, Pitt B, Davis CE, Hood WB, Cohn JN. Effect of enalapril on survival in patients with reduced left ventricular ejection fractions and congestive heart failure. SOLVD Investigators. N Engl J Med. 1991;325(5):293-302.
• ²
  CONSENSUS Trial Study Group. Effects of enalapril on mortality in severe congestive heart failure. Results of the Cooperative North Scandinavian Enalapril Survival Study (CONSENSUS). N Engl J Med. 1987;316(23):1429-35.
• ³
  Connolly SJ, Ezekowitz MD, Yusuf S, Eikelboom J, Oldgren J, Parekh A, et al. Dabigatran versus warfarin in patients with atrial fibrillation. N Engl J Med. 2009;361(12):1139-51.
• ⁴
  Granger CB, Alexander JH, McMurray JJ, Lopes RD, Hylek EM, Hanna M, et al. Apixaban versus warfarin in patients with atrial fibrillation. N Engl J Med. 2011;365(11):981-92.

Publication Dates

• Publication in this collection
  Oct-Dec 2018