Introduction

Meta-analysis is a method used to combine the results of different trials in order to obtain a quantified synthesis. The size of individual clinical trials is often too small to detect treatment effects reliably. Meta-analysis is a way to increase the power of statistical analyses by pooling the results of all available trials. This can be done by using summarized results of published trials or individual patient data obtained from all existing trials (published or not).
As we are trying to use the meta-analysis to estimate a combined effect from a group of similar studies, we need to check that the effects found in the individual studies are similar enough that we are confident a combined estimate will be a meaningful description of the set of studies. In doing this, we need to remember that the individual estimates of treatment effect will vary by chance, because of randomization. So we expect some variation. What we need to know is whether there is more variation than we'd expect by chance alone. When this excessive variation occurs, we call it statistical heterogeneity, or just heterogeneity.

Identifying statistical heterogeneity

You can determine the presence of statistical heterogeneity in two main ways:

1) Cochrane's χ ² (chi-square) test

A more common way of indicating the extent of heterogeneity is a statistical test, often described as Cochran's χ 2 test or the Q-test. This test is based on the squared difference between the estimated treatment effect in trial i (T i ) and the overall estimated treatment effect (T) weighted by the inverse of the estimated variance of the treatment effect in trial i (w i ).

Where k is the number of studies being combined, and:

Under the null hypothesis of homogeneity, this statistic follows a chi-square distribution with K-1 degrees of freedom (d.f.). When the value of the Q-statistic is too high (that is, overall, the trial effects are too far from the mean effect, taking into account the sampling variance), the null hypothesis of homogeneity is rejected. This test for heterogeneity is considered to have low power.

2) Graphical methods

1. Plot of normalized (z) score
2. Forest Plot
3. Radial Plot (Galbraith diagram)
4. L'Abblé plot

Possible cause of heterogeneity

We are interested in detecting factors that may produce variation in the effects of treatments. These factors may be categorized as:

1. due to chance;
2. due to the scale used to measure the treatment effect;
3. due to treatment characteristic which can be investigated;
4. characteristic of the design and conduct of the studies;

Specific factors that may cause heterogeneity

In the sections below, we discuss how certain factors from the above list may affect heterogeneity:

• Impact of underlying risk on heterogeneity: If the intervention leads to a constant proportional reduction in effect, then the level of risk or severity of disease is important for predicting absolute effects. In this instance, high-risk patients or those with more severe disease will tend to have a greater absolute benefit.
  
• Impact of size of intervention dose on heterogeneity: Many interventions will have a dose–response relationship: the effects increase with increasing intensity of the intervention (in the form of increased dose, increased duration or increased frequency). The effect will often plateau once an optimal dosage is reached and may even become harmful beyond this. Hence, exploration of such dose–response relationships is always advisable in any systematic review. It should be noted, however, that the dose–response relationship may be different for different outcomes.
  The timing of therapy may also be important. A classic example is that of thrombolytic therapy for acute myocardial infarction, for which the benefits rapidly decline with time since the onset of chest pain.
  
• Length of follow-up: the length of follow-up of a trial may have an influence on the estimate of treatment effect. The following issues need to be considered when investigating the impact of this factor: (a) treatment effects may vary over time; (b) trials with longest follow-up may be selective because they may have been designed and conducted earlier; and (c) a summary measurement based on an overall risk reduction that assume constant annual risk ratios may differ from actuarial estimates based on yearly assessment. A possible solution to this problem would be to use a survival type analysis.
  
• Co-interventions: Other treatments may have an effect on the intervention under study. For example, the antihypertensive effects of the angiotensin-converting enzyme inhibitor enalapril vary according to whether a patient is also taking a beta-blocker. This is due to a significant negative interaction for the hypotensive effect that exists when both agents are taken simultaneously.

Methods for investigating and dealing with heterogeneity

If you identify or suspect that important diversity or heterogeneity is present in your review, there a several options open to you. Don't forget that one option is that of not performing a meta analysis. An unwise meta-analysis can lead to highly misleading conclusions. In the remainder of this section we take a brief look at some options for investigating or incorporating heterogeneity in a review.

1) Change scale of outcome variable

It may be sufficient to simply change the scale on which the study outcomes are measured, to remove heterogeneity. For binary outcomes, changing the outcome from a absolute measure to a relative measure can reduce the degree of heterogeneity. For continuous outcomes, a transformation such as taking logarithms is common practice, though there may be a trade off between statistical homogeneity and clinical interpretability.

2) Meta regression

Meta regression is a collection of statistical procedures to assess heterogeneity, in which the effect size of study i is regressed on one or several covariates, with a value defined for each study.

3) Use of random effects models

When there are substantial differences among trial results, and in the face of statistical heterogeneity, a single estimate may be misleading and should be avoided. However, a strategy considered is the use of the random effects model in the calculation of a pooled estimator that formally takes into consideration the heterogeneity among the trial results.

4) Subgroup analysis

If there were some types of participant, intervention or outcome you thought were likely to be quite different to the others, you might plan a subgroup analysis.

References

1. Sutton A.J., Abrams K.R., Jones D.R., Sheldon T.A.,Song F. Methods for meta analysis in medical research: Assessing Between Study Heterogeneity. 2002
2. Egger M, Davey Smith G. Meta-analysis: potentials and promise. BMJ 1997; 315: 1371-4
3. Villar J, Eugenia Mackey M, Carroli G, Donner A. Meta-analyses in systematic reviews of randomized controlled trials in perinatal medicine: comparison of fixed and random effects models. Statistics in Medicine 2001;20:3635-3647
4. Baujat B, Mahe C, Pignon J.P., Hill C. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Statistics in Medicine 2002;
   20:3635–3647
5. Glasziou P.P., Sanders S.L. Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine 2002; 21:1503–1511
6. Cochrane Collaboration open learning material for reviewers: Module 13: Diversity and heterogeneity. November 2002
7. Delgado-Rodríguez M. Glossary on meta-analysis

Behnam Shakiba

Top of the page