top of page
Search

Revision of Reporting Guidelines

Dear fellow meta-analysts:


It has been 7 years since we drafted the current reporting guidelines. I believe the time is ripe for updating them – and based on our debate at the MAER-Net Open Forum in Greenwich, I think at least some of you share that belief. Let me kick off a discussion on the shape of the possible revision.


The intention is not to create a list of rules and then cast them in stone. Rather, the guidelines should, apart from defining the minimum standard for conducting a meta-analysis in economics (well covered by the current guidelines), offer a set of practical recommendations. These recommendations may change again in a couple of year as our tools evolve.


My motivation for proposing the update is that since I started to serve as an associate editor at the Journal of Economic Surveys, I’ve handled quite a few meta-analyses with basic econometric and interpretation errors, studies that do not enhance the reputation of meta-analysis in economics. I find myself providing similar feedback all over again (and the same, I assume, goes for Tom and Chris at the JoES and many of you who write referee reports), so I believe a concrete set of recommendations published in the JoES will help.


Below I offer 12 subjective recommendations that I miss in the current guidelines. You may disagree about some points or may want to add others. We will prepare the revision of the guidelines (if any) based on the discussion that will follow. Thank you for contributing!


  1. Weights. If the meta-analyst doesn't use inverse-variance weights, she or he should explain why. The requirement is in line with the recent review paper on meta-analysis in Nature: “Meta-analyses that are not weighted by inverse variances are common and often poorly justified.”

  2. Outliers. The meta-analyst is encouraged to specify how outliers, both in the estimated effects and standard errors, are treated: whether all observations are included, or which rule was used to omit outlying observations (for example, Hadi or winsorizing and the respective thresholds).

  3. Reconstructed standard errors. If standard errors are not directly reported in some primary studies, the meta-analyst should state how the standard errors were obtained (for example, using the delta method with the assumption of zero covariance). A robustness check is encouraged that excludes observations with reconstructed standard errors.

  4. Study-level dummies. When using meta-regression analysis to investigate the extent of publication bias, the meta-analyst is encouraged to include a robustness check with study-level dummies (fixed effects in the econometric sense) and thus control for unobserved characteristics of individual studies. Note that such specification only captures within-study bias (p-hacking).

  5. Random effects. Study-level random effects in economics meta-analyses can be correlated with publication bias or other aspects of studies. The meta-analyst should exercise caution when adding random effects to multiple MRA, because doing so likely violates the exogeneity condition.

  6. Clustering or bootstrapping. Standard errors in meta-regression analysis should be clustered or bootstrapped. Bootstrapping is the only viable option when the number of studies is small.

  7. General-to-specific. Instead of sequential t-tests, in multiple MRA it is recommended to use the more holistic general-to-specific approach due to Hendry and colleagues. Alternatively, the meta-analyst may want to employ Bayesian or frequentist model averaging to address model uncertainty.

  8. Sensitivity of model averaging. If the meta-analyst uses Bayesian or frequentist model averaging, she or he should report robustness checks that show how the results depend on the selected priors (in the Bayesian case) or the selected weights (Mallows or other; in the frequentist case). The procedure employed to simplify model space (Markov Chain Monte Carlo or orthogonalization) should be mentioned.

  9. Collinearity. The meta-analyst is encouraged to report collinearity statistics for multiple MRA, for example the correlation matrix or variance-inflation factors. Note that collinearity increases when inverse-variance weights are used.

  10. Robustness checks. The meta-analyst should report robustness checks to the baseline test of publication bias and the underlying effect. Note that different estimators have different performance in different environments (as shown by Carter et al.). Choose several robustness checks, for example: PET-PEESE, WAAP, Bom & Rachinger, Furukawa, Hedges (and variants thereof), Andrews & Kasy, p-curve, p-uniform.

  11. Data. The meta-analyst is encouraged to provide the data to the editor and referees so that they can check their structure. This can be done either through the journal's submission system or (preferably) publicly through the author's website.

  12. Economic significance. The meta-analyst should discuss the economic significance of results. For example, publication bias or the underlying effect can be significant statistically, but not material in practice. If partial correlation coefficients are used, Doucouliagos's guidelines for the practical strength of the effect should be consulted.

493 views13 comments

Recent Posts

See All

Methods Guidelines for Meta-Analysis

Zuzana Irsova (Charles University & Anglo-American University in Prague) As discussed at the Palma colloquium, together with Tom, Chris, and Tomas we prepared a brief practical guide on how to do mode

13 Comments


Tom Stanley
Tom Stanley
Dec 19, 2019

Dear MAER-Net members and friends.


Thanks greatly for your interest and participation.


It seems that this discussion has reached a natural conclusion. I perceive a consensus that revisions are merited even if they are rather modest. So be it. We invite everyone who took the time to make a comment above to be a coauthor of an associated Journal of Economics Surveys paper that we hope results from this discussion.


After the New Year, we intend to initiate a new discussion about ‘best practices’ for MRA in economics. So, please check this blog from time to time and offer a new blog post about one of your research projects.


Warm Holiday Greetings and a Happy New Year!


Tom

Like

Tom Stanley
Tom Stanley
Dec 03, 2019

Dear Colleagues:

Thanks greatly for everyone’s contribution and attention. This has been an interesting conversation. Below is my attempt to revise the MAER-Net reporting guidelines to incorporate many, but not all, of the suggestions and comments made. I marked the changes in bold and italics, please compare to the original, https://onlinelibrary.wiley.com/doi/epdf/10.1111/joes.12008

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Reporting Guidelines for Meta-Regression Analyses in Economics

Research papers that conduct meta-regression analyses (MRA) in economics should include:


Research Questions and Effect Size

• A clear statement of the specific economic theories, hypotheses, or effects studied.

• A precise definition of how effects are measured (the ‘effect size’) and their standard errors, accompanied by any relevant formulas if transformations are made.

• An explicit description about how measured effects are comparable,…


Like

Wow! This has been an excellent discussion. I have learned a lot. Thanks to Tomas for the initiative, and everybody who has contributed thoughtful points. Maybe what we should do is require that anybody who submits a meta-analysis to a journal should tick a box indicating that they have read this discussion!


I think Tom makes a strong argument that whatever we come up with would be better presented as "recommendations" rather than "guidelines".


I also think Tom's point about providing a PRISMA diagram is something that should be added to the list. I haven't done this in the past, and I realize now that I should be doing this.


Here are some other possible things to add to the…



Like

Dear All,


I greatly appreciate all the comments and issues raised and please, do not take my comment as in any way dismissive of these very important issues. However, my sense is that the origin of meta-analysis in general and MAERnet in particular is what we could somewhat pompously call "true and credible science". The technical part is of course of paramount importance, but a perfectly executed empirical endeavor is not yet a paper, let alone a project. To me, the most fascinating part of MAERnet is the focus on academic significance of the issues analyzed, the policy relevance of the findings -- and real quest for truth behind it. The driving force behind MAERnet movement is not to run…


Like

Ichiro
Ichiro
Nov 23, 2019

Dear Tomas and MAER-Net members!


I basically agree with the idea to revise the Reporting Guidelines now.


What I would like to stress in this regard is that the Guidelines should provide minimum conditions to enable the third party to check or reproduce a meta-analysis in a paper in question to secure "replicability" of the reported results as Mullen (1989) stressed in his book entitled "Advanced BASIC Meta-analysis" (pp. 7-8).


In other words, the Guidelines should not be so restrictive that meta-analysts are curbed freedom of methodological choice. Hence I am not very positive to force researchers to follow a certain procedure and/or methodology of meta-analysis in the Guidelines even if it is correct.


At the same time, however, I…


Like
bottom of page