[R-meta] rma, sandwich correction and very small data sets

Viechtbauer, Wolfgang (SP) wo||g@ng@v|echtb@uer @end|ng |rom m@@@tr|chtun|ver@|ty@n|
Fri Dec 11 22:03:30 CET 2020

Just want to add to this an answer to question 5.

robust() from metafor with adjust=TRUE does use a small-sample adjustment, but a very simple one. clubSandwich has much more sophisticated adjustments and I would recommend to use them (CR2 is the recommended one).


>-----Original Message-----
>From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>Sent: Wednesday, 09 December, 2020 17:47
>To: Valeria Ivaniushina; Viechtbauer, Wolfgang (SP)
>Cc: R meta
>Subject: Re: [R-meta] rma, sandwich correction and very small data sets
>Dear Valeria
>I think as a general principle you are entitled to do your analysis even
>on a small data-set as long as you accept that your results may not be
>very precise. There seems to be a general feeling among analysts in the
>area in which I work (health) that looking for small study effects is
>not worth trying with fewer than ten studies and even with more may well
>be uninformative. I am personally rather sceptical about identifying
>observations as outliers in the absence of a scientific reason for doing so.
>On 09/12/2020 15:21, Valeria Ivaniushina wrote:
>> Dear Wolfgang,
>> Thank you VERY much!
>> Thank you for correcting my code -- indeed, random effect on the 1st level
>> is totally needed!
>> A couple more questions, if I may
>> 1. There are too little cases for such a complex data structure, and it's
>> serious limitation.
>> But I hope that even if the results may be considered only as descriptive,
>> they still point out in the correct direction?
>> Especially taking into account that all three subsamples show quite
>> results.
>> Is it a valid interpretation?
>> 2. Considering that the sample is small (and 3-level!),  I guess that
>> analysis of outliers would be excessive. Is it right?
>> 3. The same goes for publication bias analysis? (as James points out,
>> tests do not have strong power:
>> www.jepusto.com/publication/selective-reporting-with-dependent-effects/ )
>> 4. and there is no power for mediation analysis, so I don't have to even
>> attempt to do it?
>> 5. Estimators question:
>>   "robust" function in rma is using sandwich-type estimator, and with
>> = TRUE it does a small-sample adjustment
>> In the clubSandwich library there are a bunch of estimators with different
>> small sample corrections. They give somewhat different results, some are
>> very close to "robust" output
>> Is clubSandwich  CR2 (for example) better than robust.rma?
>> Or, if CR estimators from clubSandwich are not definitely preferable, can
>> just use robust.rma?
>> Best,
>> Valeria

More information about the R-sig-meta-analysis mailing list