[R-meta] RVE with rma: different heterogeneity for different values of the constant correlation

[Andreas Voldstad]

Dear meta-analysts,

I am conducting a meta-analysis of standardised mean differences with many sources of dependency (multiple outcomes measuring the same construct, multiple control groups, outcomes from dyads (e.g., separate patient-carer and husband-wife scores), and multiple timepoints (post and followups)).

I used metafor::rma.mv to construct 3-level models (random = ~1|study/effectsize) in combination with RVE (metafor::robust). I have conducted sensitivity analyses with different assumptions for the constant correlation rho.

As expected, the pooled effect and standard error is nearly identical across a range of values of rho, with the same inference of a significant effect.

However, the variance components and heterogeneity are not affected by RVE and are different for different values of the correlation.

For the full dataset, which includes an extreme outlier study, which also has extreme differences between the effect sizes within the study, total I2 was practically identical across values of rho, but the levels varied. I2 level 3 was inversely proportional to rho, and varied from 80.45 to 88.42.

Removing the outlier, total I2 varied from 40.73, 95% CI = [3.47, 73.86] to 55.92 [24.04, 79.38], with I2 level 3 from 6.81 to 55.92. Q varied, with some models showing highly significant heterogeneity (e.g., p<.001) and some models showing non-significant heterogeneity (p>.05).

Based on this, I have the following questions:

Question 1:
It seems I do not know how large the proportion of heterogeneity actually is, and it is sensitive to my imputed constant correlation. I was wondering if you have any suggestions regarding cluster-robust inferences on heterogeneity.

Question 2:
I was wondering if the ICC that can be calculated after fitting the model can be used as an indication of how "right" the initial guess of the constant correlation is?

E.g., rma_mv_model$sigma2[1]/ sum(rma_mv_model$sigma2)
or the "rho" value that is produced by rma.mv when reparametrizing the model as random = ~factor(effectsize)|Study

Based on a suggestion from James Pustejovsky, I  compared the log likelihood and other information criteria of models fitted with different values for rho. For the full dataset with the extreme outlier study, I got the curious result that the loglikelihood was better at lower values of rho and the best-fitting model was the one with the smallest rho tested (.05).

Removing the outliers, a reasonable value of rho (.5) gave the best fit. This was smaller than my initial guess of .7. However, that guess was based on known correlations between partners in dyads, correlations between outcomes, correlations between timepoints.

There are other pairs of effects that might be expected to be less correlated than this (e.g., cases such as: the correlation between partner A's effect size in comparison with control group A at time 2, and partner B's effect size in comparison to control group B at time 3.)

Question 3:
As mentioned, some correlations are known, at least from some large studies in the dataset. For studies with multiple control groups, I know there are ways to calculate the covariance between effect sizes using the sample size. However, I am not sure how to build the whole covariance matrix based on this information and using clubsandwich::pattern_covariance_matrix or impute_covariance_matrix, since the dataset contains all of these sources of dependency at the same time, and sometimes within the same study (e.g., example at the end of question 2).

Is there any guidance available for this situation?


Best wishes,

Andreas Voldstad (he/him)
PhD student in Psychiatry
University of Oxford
Please don�t feel obliged to read or respond to my email outside your own working hours.


	[[alternative HTML version deleted]]