[R-meta] rma, sandwich correction and very small data sets

[Viechtbauer, Wolfgang (SP)]

Just want to add to this an answer to question 5.

robust() from metafor with adjust=TRUE does use a small-sample adjustment, but a very simple one. clubSandwich has much more sophisticated adjustments and I would recommend to use them (CR2 is the recommended one).

Best,
Wolfgang

>-----Original Message-----
>From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>Sent: Wednesday, 09 December, 2020 17:47
>To: Valeria Ivaniushina; Viechtbauer, Wolfgang (SP)
>Cc: R meta
>Subject: Re: [R-meta] rma, sandwich correction and very small data sets
>
>Dear Valeria
>
>I think as a general principle you are entitled to do your analysis even
>on a small data-set as long as you accept that your results may not be
>very precise. There seems to be a general feeling among analysts in the
>area in which I work (health) that looking for small study effects is
>not worth trying with fewer than ten studies and even with more may well
>be uninformative. I am personally rather sceptical about identifying
>observations as outliers in the absence of a scientific reason for doing so.
>
>Michael
>
>On 09/12/2020 15:21, Valeria Ivaniushina wrote:
>> Dear Wolfgang,
>>
>> Thank you VERY much!
>> Thank you for correcting my code -- indeed, random effect on the 1st level
>> is totally needed!
>>
>> A couple more questions, if I may
>>
>> 1. There are too little cases for such a complex data structure, and it's
>a
>> serious limitation.
>> But I hope that even if the results may be considered only as descriptive,
>> they still point out in the correct direction?
>> Especially taking into account that all three subsamples show quite
>similar
>> results.
>> Is it a valid interpretation?
>>
>> 2. Considering that the sample is small (and 3-level!),  I guess that
>> analysis of outliers would be excessive. Is it right?
>>
>> 3. The same goes for publication bias analysis? (as James points out,
>these
>> tests do not have strong power:
>> www.jepusto.com/publication/selective-reporting-with-dependent-effects/ )
>>
>> 4. and there is no power for mediation analysis, so I don't have to even
>> attempt to do it?
>>
>> 5. Estimators question:
>>   "robust" function in rma is using sandwich-type estimator, and with
>adjust
>> = TRUE it does a small-sample adjustment
>> In the clubSandwich library there are a bunch of estimators with different
>> small sample corrections. They give somewhat different results, some are
>> very close to "robust" output
>> Is clubSandwich  CR2 (for example) better than robust.rma?
>> Or, if CR estimators from clubSandwich are not definitely preferable, can
>I
>> just use robust.rma?
>>
>> Best,
>> Valeria