A little over a year ago I went to the MAER-Net Colloquium in Kyoto with a number of questions I wanted to have answered. In fact, I used my presentation as an opportunity to take my questions to the MAER-Net “brain trust”. Below is a slide from the presentation I gave in Kyoto:

Here is the background: My presentation was on “The Relationship Between Social Capital and Economic Growth: A Meta-Analysis.” Because measures of social capital and economic growth vary widely across studies, we transformed the estimates from the original studies into partial correlation coefficients (PCCs).

As is standard in economics, we used the following expression for the sampling variance of PCC:

1) s.e.(PCC)^2 = (1-PCC^2) / df

In the course of our analysis, one of the co-authors on this project, Robbie van Aert (Tilburg University) said we were using the wrong expression for the sampling variance of PCC. He said the correct expression was:

2) s.e.(PCC)^2 = (1-PCC^2)^2 / df

Notice the difference in the numerators.

This was pretty shocking to me, as I had published several meta-analyses with PCCs using Equation (1). As had many other economists.

Indeed, Robbie was right. Economists were using the wrong sampling variance! As a result of this experience, he published a note in Research Synthesis Methods (see below).

Unfortunately, I wasn’t able to get much of a response from those attending MAER-Net in Kyoto so I left confused about what I should do in my research.

However, the answer to my question was not long in coming. In March of this year I learned of an article by Tom Stanley and Chris Doucouliagos that addressed the issue of the “correct” sampling variance of PCC (see below).

To cut to the chase, the answer to my question if economists were using the wrong s.e.(PCC) is twofold:

1) Yes, economists are using the wrong sampling variance of PCC

2) Economists should continue using the wrong sampling variance because it produces better estimates

The reason why the “wrong” sampling variance is better than the “correct” sampling variance is enlightening. It raises issues about PCCs that I never appreciated. I thought if I was unaware of these issues, maybe others were too. Hence the motivation for this blog.

First, a reminder about why meta-analysis uses inverse variance weighting. Given heteroskedasticity, it is well known that weighted least squares (WLS) will produce estimates with least variance. Ceteris paribus, that argues in favor of using the “correct” sampling variance of PCC.

However, ceteris paribus doesn’t hold because the “correct” sampling variance of PCC is a function of PCC (see Equation 1).

In particular, as PCC increases, s.e.(PCC) decreases. As a result, inverse variance weighting favors larger values of PCC. This introduces bias in the estimation of the overall mean.

Doesn’t the “wrong” sampling variance also have a bias problem (cf. Equation 2)? Yes, it does. But the bias is not as bad.

Stanley and Doucouliagos demonstrate this in a series of simulations reported in their paper. In the table below, S1^2 is the “correct” sampling variance, and S2^2 is the “wrong” sampling variance commonly used by economists. In every case, bias is less using S2^2.

Does that mean that somehow WLS isn’t relevant for PCCs? Not at all. It is still the case that inverse weighting with the “correct” sampling variance produces estimates with smaller variance.

However, its advantage in variance is outweighed by its disadvantage in bias. As a result, inverse variance weighting using the “wrong” sampling variance is more efficient. This is demonstrated in the table where root mean squared error (RMSE) for S2^2 is smaller than for S1^2.

In summary, the “correct” sampling variance of PCC produces estimates with smaller variance. The “wrong” sampling variance produces estimates with smaller bias.

As Stanley and Doucouliagos show, the “wrong” sampling variance makes a better bias-variance trade-off and is thus more efficient. Accordingly, they recommend that economists continue to use the “wrong” sampling variance of PCC in inverse variance weighting.

Before reading Stanley and Doucouliagos’ article, I was unaware that inverse variance weighting with PCCs involved a bias-variance trade-off. Perhaps others were also unaware and will find this blog useful.

REFERENCES

Stanley, T. D., & Doucouliagos, H. (2023). Correct standard errors can bias meta‐analysis. Research Synthesis Methods, 14(3), 515-519.

van Aert, R. C., & Goos, C. (2023). A critical reflection on computing the sampling variance of the partial correlation coefficient. Research Synthesis Methods, 14(3), 520-525.

Hi Bob,

Nice summary indeed! I think it is good to emphasize that meta-analyzing PCCs has the potential to also distort the statistical inferences of the FAT test/Egger's regression test (and any variants thereof) due to the dependence between the PCC and its sampling variance/standard error.

The option of meta-analyzing Fisher's z transformed PCCs rather than PCCs directly is also worth mentioning. This removes the dependence between the PCC and its sampling variance, because the sampling variance of the Fisher's z transformed PCC is not a function of the (Fisher's z transformed) PCC. Another advantage is that meta-analyzing the Fisher's z transformed PCCs is more in line with the assumptions of the meta-analysis models, because the sampling distribution of the…

Hi Bob, thanks for a nice summary. I would add:

1) The bias in meta-analysis due to mechanical dependence between PCCs and their standard errors is pretty small for sample sizes typical in economics.

2) Intuitively, it often makes more sense to prefer a more conservative measure of precision. See, for example, http://www.stat.columbia.edu/~gelman/research/published/standarderror.pdf. For me that's another reason why to prefer the "wrong" SE of PCC.

3) Many effect sizes used in economics have the same problem (mechanical correlation between effects and standard errors). For example, SMD. Our new MAIVE estimator also addresses these issues.

Be well,

Tomas