[R-meta] rma, sandwich correction and very small data sets

[Viechtbauer, Wolfgang (SP)]

Dear Valeria,

Unless you have very good reasons to assume that estimates within studies are homogeneous, you should always add a random effect at the estimate level to the model. See:

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011

and esp. the "A Common Mistake in the Three-Level Model" section.

So, I would do:

wb$ID_estimate <- 1:nrow(wb)

random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 | ID_database)

Also, if you use data=wb, you do not need wb$ in the model call.

Finally, SE_Influence sounds like this is a variable for the standard errors. The second argument of rma.mv() is for specifying the sampling *variances* (or an entire var-cov matrix).

So, to summarize:

eff1 <- rma.mv(yi=EFFECT_SIZE_Influence, V=SE_Influence^2, 
               random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 | ID_database), 
               tdist=TRUE, data=wb)

However, with the number of levels you show, I would indeed be worried about fitting such a complex model with so little data. You won't get precise estimates of the variance components and hence they can be all over the place.

Also, cluster-robust inference methods work asymptotically, that is, when the number of levels of the clustering variable gets large. With 5 or 7 levels for 'databases', I would say we are rather far away from 'asymptotically'. The clubSandwich package you are using for this includes small-sample corrections which should help a but, but I would still question the use of such methods with such low k at the clustering level. Maybe James Pustejovsky (the author of clubSandwich) can chime in here.

As for combining the results of multiple (independent) meta-analyses, see:

http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On Behalf Of Valeria Ivaniushina
>Sent: Monday, 07 December, 2020 18:00
>To: R meta
>Subject: [R-meta] rma, sandwich correction and very small data sets
>
>Dear colleagues,
>
>I do 3-level meta-analysis with a small number of studies and a small
>number of clusters.
>1st level - model, 2nd level - study, 3rd level - database.
>
>The effect I am interested in can be specified in different ways. Experts
>in the field advised me to make separate meta analyses for each
>specification and then combine the results, kind of meta-meta.
>
>I have several questions:
>
>1)  Is this a correct code?
>First I do REML:
>eff1 <- rma.mv(yi=wb$EFFECT_SIZE_Influence,
>V=wb$SE_Influence,random = list(~1 | ID_study, ~1 | ID_database),
>tdist=TRUE, data=wb)
>
>Then with this object I use sandwich, to get cluster-robust standard
>errors, clustering at the highest level of nesting:
>coef_test(eff1, vcov = "CR2",cluster = wb$ID_database)
>
>2)  I am worried that the numbers of clusters are too small -- are the
>results reliable?
>eff1: 17 models, 12 studies, 5 databases
>eff2: 8 models, 5 studies, 5 databases
>eff3: 11 models, 9 studies, 7 databases
>
>3) Variance distribution is vastly different between three models - what
>does it tell me?
>eff1: 1st level 4%, 2nd level 0%, 3rd level 96%
>eff2: 1st level 100%, 2nd level 0%, 3rd level 0%
>eff3: 1st level 15%, 2nd level 0%, 3rd level 85%
>
>4) How can I combine the results of three meta-analyses?
>
>Best,
>Valeria