Editing Meta-analysis (section)

==== Random effects model ====

Most meta-analyses are based on sets of studies that are not exactly identical in their methods and/or the characteristics of the included samples.<ref name=":5" /> Differences in the methods and sample characteristics may introduce variability (“heterogeneity”) among the true effects.<ref name=":5" /><ref>{{Cite journal |last1=Holzmeister |first1=Felix |last2=Johannesson |first2=Magnus |last3=Böhm |first3=Robert |last4=Dreber |first4=Anna |last5=Huber |first5=Jürgen |last6=Kirchler |first6=Michael |date=2024-08-06 |title=Heterogeneity in effect size estimates |journal=Proceedings of the National Academy of Sciences |language=en |volume=121 |issue=32 |pages=e2403490121 |doi=10.1073/pnas.2403490121 |issn=0027-8424 |pmc=11317577 |pmid=39078672|bibcode=2024PNAS..12103490H }}</ref> One way to model the heterogeneity is to treat it as purely random. The weight that is applied in this process of weighted averaging with a random effects meta-analysis is achieved in two steps:<ref>{{cite journal | vauthors = Senn S | title = Trying to be precise about vagueness | journal = Statistics in Medicine | volume = 26 | issue = 7 | pages = 1417–1430 | date = March 2007 | pmid = 16906552 | doi = 10.1002/sim.2639 | s2cid = 17764847 | doi-access = free }}</ref>

# Step 1: Inverse variance weighting
# Step 2: Un-weighting of this inverse variance weighting by applying a random effects variance component (REVC) that is simply derived from the extent of variability of the effect sizes of the underlying studies.

This means that the greater this variability in effect sizes (otherwise known as [[study heterogeneity|heterogeneity]]), the greater the un-weighting and this can reach a point when the random effects meta-analysis result becomes simply the un-weighted average effect size across the studies. At the other extreme, when all effect sizes are similar (or variability does not exceed sampling error), no REVC is applied and the random effects meta-analysis defaults to simply a fixed effect meta-analysis (only inverse variance weighting).

The extent of this reversal is solely dependent on two factors:<ref name="ReferenceA">{{cite journal | vauthors = Al Khalaf MM, Thalib L, Doi SA | title = Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses | journal = Journal of Clinical Epidemiology | volume = 64 | issue = 2 | pages = 119–123 | date = February 2011 | pmid = 20409685 | doi = 10.1016/j.jclinepi.2010.01.009 }}</ref>

# Heterogeneity of precision
# Heterogeneity of effect size

Since neither of these factors automatically indicates a faulty larger study or more reliable smaller studies, the re-distribution of weights under this model will not bear a relationship to what these studies actually might offer. Indeed, it has been demonstrated that redistribution of weights is simply in one direction from larger to smaller studies as heterogeneity increases until eventually all studies have equal weight and no more redistribution is possible.<ref name="ReferenceA"/>
Another issue with the random effects model is that the most commonly used confidence intervals generally do not retain their coverage probability above the specified nominal level and thus substantially underestimate the statistical error and are potentially
overconfident in their conclusions.<ref name="Brockwell2001">{{cite journal | vauthors = Brockwell SE, Gordon IR | title = A comparison of statistical methods for meta-analysis | journal = Statistics in Medicine | volume = 20 | issue = 6 | pages = 825–840 | date = March 2001 | pmid = 11252006 | doi = 10.1002/sim.650 | s2cid = 16932514 }}</ref><ref name="Noma2011">{{cite journal | vauthors = Noma H | title = Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections | journal = Statistics in Medicine | volume = 30 | issue = 28 | pages = 3304–3312 | date = December 2011 | pmid = 21964669 | doi = 10.1002/sim.4350 | hdl-access = free | hdl = 2433/152046 | s2cid = 6556986 }}</ref> Several fixes have been suggested<ref>{{cite journal | vauthors = Brockwell SE, Gordon IR | title = A simple method for inference on an overall effect in meta-analysis | journal = Statistics in Medicine | volume = 26 | issue = 25 | pages = 4531–4543 | date = November 2007 | pmid = 17397112 | doi = 10.1002/sim.2883 | s2cid = 887098 }}</ref><ref>{{cite journal | vauthors = Sidik K, Jonkman JN | title = A simple confidence interval for meta-analysis | journal = Statistics in Medicine | volume = 21 | issue = 21 | pages = 3153–3159 | date = November 2002 | pmid = 12375296 | doi = 10.1002/sim.1262 | s2cid = 21384942 }}</ref> but the debate continues on.<ref name="Noma2011" /><ref name="pmid19016302">{{cite journal | vauthors = Jackson D, Bowden J | title = A re-evaluation of the 'quantile approximation method' for random effects meta-analysis | journal = Statistics in Medicine | volume = 28 | issue = 2 | pages = 338–348 | date = January 2009 | pmid = 19016302 | pmc = 2991773 | doi = 10.1002/sim.3487 }}</ref> A further concern is that the average treatment effect can sometimes be even less conservative compared to the fixed effect model<ref>{{cite journal | vauthors = Poole C, Greenland S | title = Random-effects meta-analyses are not always conservative | journal = American Journal of Epidemiology | volume = 150 | issue = 5 | pages = 469–475 | date = September 1999 | pmid = 10472946 | doi = 10.1093/oxfordjournals.aje.a010035 | doi-access = free }}</ref> and therefore misleading in practice. One interpretational fix that has been suggested is to create a prediction interval around the random effects estimate to portray the range of possible effects in practice.<ref>{{cite journal | vauthors = Riley RD, Higgins JP, Deeks JJ | title = Interpretation of random effects meta-analyses | journal = BMJ | volume = 342 | pages = d549 | date = February 2011 | pmid = 21310794 | doi = 10.1136/bmj.d549 | s2cid = 32994689 }}</ref> However, an assumption behind the calculation of such a prediction interval is that trials are considered more or less homogeneous entities and that included patient populations and comparator treatments should be considered exchangeable<ref name="pmid23494781">{{cite journal | vauthors = Kriston L | title = Dealing with clinical heterogeneity in meta-analysis. Assumptions, methods, interpretation | journal = International Journal of Methods in Psychiatric Research | volume = 22 | issue = 1 | pages = 1–15 | date = March 2013 | pmid = 23494781 | pmc = 6878481 | doi = 10.1002/mpr.1377 }}</ref> and this is usually unattainable in practice.

There are many methods used to estimate between studies variance with restricted maximum likelihood estimator being the least prone to bias and one of the most commonly used.<ref>{{Cite journal |last1=Langan |first1=Dean |last2=Higgins |first2=Julian P.T. |last3=Jackson |first3=Dan |last4=Bowden |first4=Jack |last5=Veroniki |first5=Areti Angeliki |last6=Kontopantelis |first6=Evangelos |last7=Viechtbauer |first7=Wolfgang |last8=Simmonds |first8=Mark |date=2019 |title=A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses |journal=Research Synthesis Methods |language=en |volume=10 |issue=1 |pages=83–98 |doi=10.1002/jrsm.1316 |pmid=30067315 |s2cid=51890354 |issn=1759-2879|doi-access=free |hdl=1983/c911791c-c687-4f12-bc0b-ffdbe42ca874 |hdl-access=free }}</ref> Several advanced iterative techniques for computing the between studies variance exist including both maximum likelihood and restricted maximum likelihood methods and random effects models using these methods can be run with multiple software platforms including Excel,<ref name="Manual">{{cite web |title=MetaXL User Guide |url=http://www.epigear.com/index_files/MetaXL%20User%20Guide.pdf |access-date=2018-09-18}}</ref> Stata,<ref name="metaan">{{cite journal|url=https://www.researchgate.net/publication/227629391|title=Metaan: Random-effects meta-analysis| vauthors = Kontopantelis E, Reeves D |date=1 August 2010|journal=Stata Journal|volume=10|issue=3|pages=395–407|via=ResearchGate |doi= 10.1177/1536867X1001000307 |doi-access=free}}</ref> SPSS,<ref>{{Cite journal |last1=Field |first1=Andy P. |last2=Gillett |first2=Raphael |date=2010 |title=How to do a meta-analysis |url=http://doi.wiley.com/10.1348/000711010X502733 |journal=British Journal of Mathematical and Statistical Psychology |language=en |volume=63 |issue=3 |pages=665–694 |doi=10.1348/000711010X502733|pmid=20497626 |s2cid=22688261 }}</ref> and R.<ref name=":5">{{Cite journal |last=Viechtbauer |first=Wolfgang |date=2010 |title=Conducting Meta-Analyses in R with the metafor Package |url=http://www.jstatsoft.org/v36/i03/ |journal=Journal of Statistical Software |language=en |volume=36 |issue=3 |doi=10.18637/jss.v036.i03 |s2cid=15798713 |issn=1548-7660|doi-access=free }}</ref>

Most meta-analyses include between 2 and 4 studies and such a sample is more often than not inadequate to accurately estimate [[study heterogeneity|heterogeneity]]. Thus it appears that in small meta-analyses, an incorrect zero between study variance estimate is obtained, leading to a false homogeneity assumption. Overall, it appears that heterogeneity is being consistently underestimated in meta-analyses and sensitivity analyses in which high heterogeneity levels are assumed could be informative.<ref name=KontopantelisP1>{{cite journal | vauthors = Kontopantelis E, Springate DA, Reeves D | title = A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses | journal = PLOS ONE | volume = 8 | issue = 7 | pages = e69930 | year = 2013 | pmid = 23922860 | pmc = 3724681 | doi = 10.1371/journal.pone.0069930 | veditors = Friede T | doi-access = free | bibcode = 2013PLoSO...869930K }}</ref> These random effects models and software packages mentioned above relate to study-aggregate meta-analyses and researchers wishing to conduct individual patient data (IPD) meta-analyses need to consider mixed-effects modelling approaches.<ref name="ipdforest">{{cite journal |url= https://www.researchgate.net/publication/257316967 |title=A short guide and a forest plot command (ipdforest) for one-stage meta-analysis| vauthors = Kontopantelis E, Reeves D |date=27 September 2013|journal=Stata Journal|volume=13|issue=3|pages=574–587 |via= ResearchGate |doi=10.1177/1536867X1301300308 |doi-access=free}}</ref>/