How econometrics informs medical research
Simple econometrics weighted least squares (UWLS) is shown to represent 600,000 medical research findings better than conventional meta-analysis models—random effects (RE) and fixed effects (FE). See the recent Journal of Clinical Epidemiology paper: Stanley et al. (2023). As many of you will know, we have been promoting an alternative to conventional meta-analysis weighted averages, UWLS, that is neither fixed nor random for a l o n g time (Stanley & Jarrell, 1989; Roberts & Stanley, 2005; Stanley & Doucouliagos, 2015; 2017; with John Ioannidis in two 2017 papers, and more recently in 2022 and 2023). Our most recent paper with Chris, John, František Bartoš, Max Maier & Wim Otte investigates how well UWLS and RE fit 67,000 medical meta-analysis and finds that UWLS is overwhelmingly the best. OK, not always, but UWLS is largely more representative of medical findings (by nearly 4 to 1) than RE, and ‘substantially’ better by over 9 to 1. See.
Why?
Well, we have also known for a long time that UWLS better estimates the population mean if there is any type of publication selection bias (well, except maybe on SEs). However, publication bias is unlikely to be the main cause of UWLS’s better AIC & BIC because medical research is not as severely selected as economics—see Bartoš et al., (2023).
More likely, because medical research is often ‘excessively homogeneous’. That is, 46% of medical research findings vary less than what their estimated sampling error variances imply. Because variances can’t be negative, RE’s heterogeneity variance (tau-sq) is set to zero. In all of these cases, UWLS fits the data better than RE as it allows for ‘excess homogeneity.’ UWLS is better than RE also when there is notable heterogeneity. Many of the cases of ‘excess homogeneity' will be sampling error as typical medical MAs have only 5 estimates. Still, there is something about medical research. . . . ?
Perhaps, there is a tendency to play ‘follow the leader’ with prior studies setting a moving target? Correlated heterogeneity might be another reason for UWLS’ superior fit. When small studies are more heterogeneous than larger studies, UWLS is known to better represent findings than RE, and we have evidence of this type of correlated heterogeneity in medicine and psychology—(IntHout et al., 2015; Stanley et al., 2022). Ahhh, but that’s the subject for another post.
Tom
References:
Bartoš F et al. Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology and economics (2022).
IntHout, J., Ioannidis, J.P.A., Borm, G.F. & Goeman, J.J. Small studies are more heterogeneous than large ones: A meta-meta-analysis. Journal of Clinical Epidemiology, 68(2015), 860–869.
Ioannidis, J.P.A., Stanley, T.D. and Doucouliagos, C. (2017). “The power of bias in economics research,” The Economic Journal, 127: F236-265.
Roberts, Colin J. & Stanley, T. D. (eds.). Issues in Meta-Regression Analysis and Publication Bias in Economics, Blackwell (2005), Oxon.
Stanley, T. D and Doucouliagos, C. Meta-regression approximations to reduce publication selection bias,” Research Synthesis Methods 5 (2014), 60-78.
̶̶̶̶̶ ̶̶ ̶ Neither fixed nor random: Weighted least squares meta-analysis, Statistics in Medicine 34 (2015), 2116-27.
̶̶̶̶̶ ̶̶ ̶ Neither fixed nor random: Weighted least squares meta-regression analysis, Research Synthesis Methods 8, 19-42.
Stanley, T.D., Doucouliagos, C. and Ioannidis, J.P.A. Finding the power to reduce publication bias, Statistics in Medicine, 36 (2017): 1580-1598.
̶̶̶̶̶ ̶̶̶ Beyond random effects: When small-study findings are more heterogeneous. Advances in Methods and Practices of Psychological Science, 5 (2022): 1–11.
Stanley TD, Ioannidis JPA, Maier M, Doucouliagos, C, Otte, WM, Bartoš F. Unrestricted weighted least squares represent medical research better than random effects in 67,308 Cochrane meta-analyses. Journal of Clinical Epidemiology (2023).
Stanley, T.D. and Jarrell, S.B. Meta-Regression Analysis: A Quantitative Method of Literature Surveys. Journal of Economic Surveys, 3(1989): 161-170.