[R-meta] Wald_test - is it powerful enough?

[Viechtbauer, Wolfgang (SP)]

Dear Cátia,

To be precise: RVE does not penalize the (fixed effect) estimates at all, as they are unchanged. It only affects the SEs of the fixed effects and, when using clubSandwich, it can also use a Satterthwaite approximation for the degrees of freedom of the test statistics. 

With respect to the SEs:

The degree to which the SEs differ between those from the working model and those you obtain after using RVE depends on the degree of discrepancy between the sources of heterogeneity/dependency that are accounted for in the working model and the actual sources of heterogeneity/dependency underlying the data. If this discrepancy is large, then this will also lead to large differences between the SEs even in large samples.

However, in smaller samples, all estimates (including the SEs) will be unstable and so one might also tend to see larger differences there (but then it is less clear whether those differences really reflect discrepancies or just instabilities).

With respect to the dfs:

Depending on the structure and size of the data, the estimated dfs from the Satterthwaite approximation can also be quite small, which can have a noticeable impact on the significance of the tests (rightly so!). This should become less of an issue in larger datasets.

Best,
Wolfgang

>-----Original Message-----
>From: Cátia Ferreira De Oliveira [mailto:cmfo500 using york.ac.uk]
>Sent: Wednesday, 22 September, 2021 9:00
>To: James Pustejovsky
>Cc: Viechtbauer, Wolfgang (SP); R meta
>Subject: Re: [R-meta] Wald_test - is it powerful enough?
>
>Dear James and Wolfgang,
>
>Would you say that the RVE models are more penalising of estimates when we have
>small samples?
>I wonder if that also could explain the disparity between wald tests and rma.mv
>since for the rest of the analyses when I have bigger sample sizes the results
>seem to converge a lot more.
>
>Best wishes,
>
>Catia