[R-meta] rma, sandwich correction and very small data sets

[Michael Dewey]

Dear Valeria

I think as a general principle you are entitled to do your analysis even 
on a small data-set as long as you accept that your results may not be 
very precise. There seems to be a general feeling among analysts in the 
area in which I work (health) that looking for small study effects is 
not worth trying with fewer than ten studies and even with more may well 
be uninformative. I am personally rather sceptical about identifying 
observations as outliers in the absence of a scientific reason for doing so.

Michael

On 09/12/2020 15:21, Valeria Ivaniushina wrote:
> Dear Wolfgang,
> 
> Thank you VERY much!
> Thank you for correcting my code -- indeed, random effect on the 1st level
> is totally needed!
> 
> A couple more questions, if I may
> 
> 1. There are too little cases for such a complex data structure, and it's a
> serious limitation.
> But I hope that even if the results may be considered only as descriptive,
> they still point out in the correct direction?
> Especially taking into account that all three subsamples show quite similar
> results.
> Is it a valid interpretation?
> 
> 2. Considering that the sample is small (and 3-level!),  I guess that
> analysis of outliers would be excessive. Is it right?
> 
> 3. The same goes for publication bias analysis? (as James points out, these
> tests do not have strong power:
> www.jepusto.com/publication/selective-reporting-with-dependent-effects/ )
> 
> 4. and there is no power for mediation analysis, so I don't have to even
> attempt to do it?
> 
> 5. Estimators question:
>   "robust" function in rma is using sandwich-type estimator, and with adjust
> = TRUE it does a small-sample adjustment
> In the clubSandwich library there are a bunch of estimators with different
> small sample corrections. They give somewhat different results, some are
> very close to "robust" output
> Is clubSandwich  CR2 (for example) better than robust.rma?
> Or, if CR estimators from clubSandwich are not definitely preferable, can I
> just use robust.rma?
> 
> Best,
> Valeria
> 
> On Wed, Dec 9, 2020 at 1:28 PM Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> 
>> Dear Valeria,
>>
>> Unless you have very good reasons to assume that estimates within studies
>> are homogeneous, you should always add a random effect at the estimate
>> level to the model. See:
>>
>> http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011
>>
>> and esp. the "A Common Mistake in the Three-Level Model" section.
>>
>> So, I would do:
>>
>> wb$ID_estimate <- 1:nrow(wb)
>>
>> random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 | ID_database)
>>
>> Also, if you use data=wb, you do not need wb$ in the model call.
>>
>> Finally, SE_Influence sounds like this is a variable for the standard
>> errors. The second argument of rma.mv() is for specifying the sampling
>> *variances* (or an entire var-cov matrix).
>>
>> So, to summarize:
>>
>> eff1 <- rma.mv(yi=EFFECT_SIZE_Influence, V=SE_Influence^2,
>>                 random = list(~ 1 | ID_estimate, ~ 1 | ID_study, ~ 1 |
>> ID_database),
>>                 tdist=TRUE, data=wb)
>>
>> However, with the number of levels you show, I would indeed be worried
>> about fitting such a complex model with so little data. You won't get
>> precise estimates of the variance components and hence they can be all over
>> the place.
>>
>> Also, cluster-robust inference methods work asymptotically, that is, when
>> the number of levels of the clustering variable gets large. With 5 or 7
>> levels for 'databases', I would say we are rather far away from
>> 'asymptotically'. The clubSandwich package you are using for this includes
>> small-sample corrections which should help a but, but I would still
>> question the use of such methods with such low k at the clustering level.
>> Maybe James Pustejovsky (the author of clubSandwich) can chime in here.
>>
>> As for combining the results of multiple (independent) meta-analyses, see:
>>
>> http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:
>> r-sig-meta-analysis-bounces using r-project.org]
>>> On Behalf Of Valeria Ivaniushina
>>> Sent: Monday, 07 December, 2020 18:00
>>> To: R meta
>>> Subject: [R-meta] rma, sandwich correction and very small data sets
>>>
>>> Dear colleagues,
>>>
>>> I do 3-level meta-analysis with a small number of studies and a small
>>> number of clusters.
>>> 1st level - model, 2nd level - study, 3rd level - database.
>>>
>>> The effect I am interested in can be specified in different ways. Experts
>>> in the field advised me to make separate meta analyses for each
>>> specification and then combine the results, kind of meta-meta.
>>>
>>> I have several questions:
>>>
>>> 1)  Is this a correct code?
>>> First I do REML:
>>> eff1 <- rma.mv(yi=wb$EFFECT_SIZE_Influence,
>>> V=wb$SE_Influence,random = list(~1 | ID_study, ~1 | ID_database),
>>> tdist=TRUE, data=wb)
>>>
>>> Then with this object I use sandwich, to get cluster-robust standard
>>> errors, clustering at the highest level of nesting:
>>> coef_test(eff1, vcov = "CR2",cluster = wb$ID_database)
>>>
>>> 2)  I am worried that the numbers of clusters are too small -- are the
>>> results reliable?
>>> eff1: 17 models, 12 studies, 5 databases
>>> eff2: 8 models, 5 studies, 5 databases
>>> eff3: 11 models, 9 studies, 7 databases
>>>
>>> 3) Variance distribution is vastly different between three models - what
>>> does it tell me?
>>> eff1: 1st level 4%, 2nd level 0%, 3rd level 96%
>>> eff2: 1st level 100%, 2nd level 0%, 3rd level 0%
>>> eff3: 1st level 15%, 2nd level 0%, 3rd level 85%
>>>
>>> 4) How can I combine the results of three meta-analyses?
>>>
>>> Best,
>>> Valeria
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html