The MAER-Net

Community.

Colloquia

The MAER-Net Colloquium is the annual meeting of our group to exchange about new meta-analysis application and methodological advances.

Resources

Find reporting guidelines for meta-analysis research in economics and publications of the MAER-Net community.

Blog

The MAER-Net blog presents discussions and posts about recent research and advances in meta-analysis research.

Welcome to the Meta-Analysis of Economics Research (MAER) Network.

MAER-Net is an international network of scholars committed to improving economic science through meta-analysis. The purpose of our network is to serve as a clearinghouse for research in meta-analysis and economics and to improve the communication among scholars in this rapidly growing field.

ABOUT META-REGRESSION ANALYSIS

What is MRA?

MRA is the statistical analysis of previously reported regression results (Stanley and Jarrell, 1989). It seeks to summarize and explain the disparate empirical findings routinely reported in nearly all areas of economics. Over the last several decades, a thousand meta-analyses have been conducted in economics, with over 100 new ones appearing each year.

Introductions to meta-regression analysis can be found in Stanley and Jarrell (1989), Stanley (2001), Stanley and Doucouliagos (2012), and Doucouliagos (2016).

What have meta-analysts learned?

Publication selection inflation is found in the majority of economic research areas, and its effects are often larger than the magnitude of underlying economic phenomenon being investigated (Doucouliagos and Stanley, 2014; Ioannidis et al, 2017).

Regression model misspecification is, in fact, the principal cause for the observed variation among reported economic findings, confirming the concerns famously expressed by Leamer (1983), Summers (1991), Sala-i-Martin (1997), and others.

That environmental values may be transferred from MRA to unstudied sites (Rosenberger and Loomis, 2000; Johnston and Rosenberger, 2010; Johnston et al., 2015)

The empirical literature often contains strong evidence against widely held economic theory and contrary to conventional narrative reviews (Stanley, 2001; Stanley, 2004; Doucouliagos and Stanley, 2009). Without some objective and systematic method of literature reviewing, conventional narrative reviews can draw any conclusions their authors wish.

Economics research is highly underpowered. A survey of 64,076 economic estimates from 159 areas of research and 6,700 empirical studies finds that the median statistical power is 18%, or less (Ioannidis et al, 2017). Impotence begets bias. Typically, reported economic effects are inflated by 100% with one-third inflated by a factor of four or more.

Publication selection inflation

"Many other commentators have addressed the issue of publication bias ... All agree that it is a serious problem"
(Begg and Berlin, 1988, p. 421)

"Are all economic hypotheses false?” de Long and Lang (1992) rhetorically asked. Researchers, reviewers and editors treat ‘statistically significant’ results more favorably; hence, they are more likely to be published. Studies that find relatively small and ‘insignificant’ effects are much less likely to be published, because they may be thought to say little about the phenomenon in question. Publication selection bias is so strong that we are likely to be better off discarding 90% of the research results than to take them at face value (Stanley, Jarrell and Doucouliagos, 2010).

“(P)ublication bias is leading to a new formulation of Gresham’s law — like bad money, bad research drives out good” (Bland, 1988, p. 450).

Funnel graphs should look like this one, below, for the union-productivity literature, though they seldom do.

Many economists have turned their attention to the issue of publication selection and have used meta-regression analysis to identify and correct it.

Card, D., Krueger, A.B., 1995. Time-series minimum-wage studies: A meta-analysis. American Economic Review 85, 238-43.
Rose, A.K., Stanley, T.D., 2005. A Meta-Analysis of the effect on common currency on international trade. Journal of Economic Surveys 19, 347-65.
Stanley, T.D., 2005. Beyond publication bias. Journal of Economic Surveys 19, 309-45.
Doucouliagos, H., Paldam, M., 2006. Aid effectiveness on accumulation: A meta study. Kyklos 59, 227-54.
Disdier, A.C., Head, K., 2008. The puzzling persistence of the distance effect on bilateral trade. Review of Economics and Statistics 90, 37-44.
Doucouliagos, H., Stanley, T.D., 2009. Publication selection bias in minimum-wage research? A meta-regression analysis. British Journal of Industrial Relations 47, 406-28.
Havranek, T. 2016. Measuring intertemporal substitution: The importance of method choices and selective reporting. Journal of the European Economic Association 13, 1180-204.
Doucouliagos, H., T.D. Stanley, M. Giles., 2012. Are estimates of the value of a statistical life exaggerated? Journal of Health Economics 31, 197-206.
Viscusi, W.K., 2015. The role of publication bias in estimates of the value of a statistical life. American Journal of Health Economics 1(1), 27–52.

No other approach can cleanse the economic literature of the distorting effect of publication selection. Economists have begun to develop MRA methods that might ‘solve’ this fundamental problem of empirical science and to render this bias mostly harmless.

Stanley, T.D., 2005a. Beyond publication bias. Journal of Economic Surveys 19, 309-45.
Stanley, T.D., 2008. Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection. Oxford Bulletin of Economics and Statistics 70, 103-127.
Stanley, T.D., Doucouliagos, C., 2012. Meta-Regression Analysis in Economics and Business. Routledge.
Stanley, T.D., Doucouliagos, C., 2014, Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods 5, 60-78.
Stanley T.D. and Doucouliagos, H, 2017. Neither fixed nor random: Weighted least squares meta-regression analysis. Research Synthesis Methods 8, 19-42.
Stanley, T.D., Doucouliagos, C., Ioannidis, J.P.A., 2017. Finding the power to reduce publication bias. Statistics in Medicine 36, 1580-1598.

In an era characterized by the rapid expansion of research publications and a flood of empirical findings on any given subject, knowledge and sensible policy action are being drowned. All reviews, whether conventional or meta, are vulnerable to publication selection bias. Without some objective and balanced way to integrate this sea of results, ideology and self-serving deceit will dominate the public discussion of economic research. What we need is some objective and critical methodology to integrate diverse research findings and to reveal the nuggets of ‘truth’ that have settled to the bottom. This is precisely what Meta-Regression Analysis can do!