[R-meta] HKSJ adjusted error by hand

[Viechtbauer, Wolfgang (NP)]

Interesting. Not sure how you did the Satterthwaite approximation. Note that the df computed by clubSandwich are specifically for the cluster-robust tests, so these are not meant to be used when doing model-based inferences but if one does this, then this seems to be overly conservative. Doing a 'proper' Satterthwaite approximation for model-based inferences does work okay for a multilevel (three-level) random-effects model with a study-level and an observation-level moderator, although robust(..., clubSandwich=TRUE) still does better from what I can tell and comes with the added benefit of being better when the var-cov structure is misspecified.

Best,
Wolfgang

> -----Original Message-----
> From: Yefeng Yang <yefeng.yang1 using unsw.edu.au>
> Sent: Friday, September 27, 2024 13:32
> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
> project.org>
> Cc: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: Re: [R-meta] HKSJ adjusted error by hand
>
> Hi Wolfgang,
>
> Thank you for clarifying the misconception I made about the relationship between
> the scale factor in the K&H method and H^2. Your illustration looks pretty good.
>
> Interesting to see this adjustment does not improve the inference for
> multilevel/multivariate models. Regarding the Satterthwaite approximation to the
> dfs you mentioned in earlier email, actually I have done a little bit of
> investigation about it before. The main point I found is that using more than
> 500 meta-analyses across three disciplines, using the combination of
> Satterthwaite approximation with multilevel models (in my case, three levels)
> performs as well as the combination of cluster-robust variance estimation with
> multilevel models. I used James's clubSandwich package, which uses both robust
> error and Satterthwaite approximation of dfs for the test of coefficient. This
> makes me  think that maybe adjusting dfs is enough in terms of hypothesis
> testing? I am not aware of whether James has done any simulation studies to
> assess the separate contribution of robust errors and dfs adjustment.
>
> Best,
> Yefeng
> ________________________________________
> From: R-sig-meta-analysis <mailto:r-sig-meta-analysis-bounces using r-project.org> on
> behalf of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis <mailto:r-sig-meta-
> analysis using r-project.org>
> Sent: Friday, September 27, 2024 19:58
> To: R Special Interest Group for Meta-Analysis <mailto:r-sig-meta-analysis using r-
> project.org>
> Cc: Viechtbauer, Wolfgang (NP)
> <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: Re: [R-meta] HKSJ adjusted error by hand
>
> Just to clarify one misconception: The scale factor in the K&H method is not
> H^2. Using the traditional definition of H^2 as Q/df, the computation of H^2
> uses the weights from a fixed-effects model and the residuals are also computed
> accordingly:
>
> dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
>
> rma_fit <- rma(yi, vi, mods = ~ ablat + year, data=dat, method="FE")
> wi <- 1 / rma_fit$vi
> ri <- residuals(rma_fit)
> df <- rma_fit$k - rma_fit$p
> # these are all identical
> sum(wi * ri^2) / df
> rma_fit$H2
> rma_fit$QE / df
>
> But in the K&H method, the scale factor can be thought of as a generalized H^2
> statistic that uses the results from a random/mixed-effects model:
>
> rma_fit <- rma(yi, vi, mods = ~ ablat + year, data=dat, method="DL")
> wi <- 1 / (rma_fit$vi + rma_fit$tau2)
> ri <- residuals(rma_fit)
> df <- rma_fit$k - rma_fit$p
> # this is not the same as H^2 or Q/df
> sum(wi * ri^2) / df
> rma_fit$H2
> rma_fit$QE / df
>
> (of course then sum(wi * ri^2) / df will change depending on which tau^2
> estimator we use and also the equivalence between H2 and Q/df breaks down once
> we use a different estimator than DL, but this is a separate issue).
>
> In any case, the computation of sum(wi * ri^2) / df can be easily generalized to
> multilevel/multivariate models, leading to a K&H method for such models. Now
> that I looked at this again, the df of k-p also follow in that case from the
> distribution of quadratic forms, so it is what it is. And based on simulation
> studies I have done, this approach did not work nearly as well as it does for
> standard random/mixed-effects models, so I never bothered to write this up.
> Might be useful though to at least put this out there, so that efforts to
> improve inference methods for such models can be focused elsewhere.
>
> Best,
> Wolfgang
>
> > -----Original Message-----
> > From: R-sig-meta-analysis <mailto:r-sig-meta-analysis-bounces using r-project.org>
> On Behalf
> > Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
> > Sent: Thursday, September 26, 2024 17:47
> > To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
> > project.org>; James Pustejovsky <mailto:jepusto using gmail.com>
> > Cc: Viechtbauer, Wolfgang (NP)
> <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>
> > Subject: Re: [R-meta] HKSJ adjusted error by hand
> >
> > I have played around with this idea and this is even implemented in rma.mv():
> >
> > dat <- dat.konstantopoulos2011
> > rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, test="knha")
> >
> > As you will see in the warning message, this is experimental. And based on
> some
> > simulations I have done, this doesn't actually work that great. Maybe this
> could
> > be improved by using a Satterthwaite approximation to the dfs. But even then,
> I
> > am not sure if this will do all that much better than:
> >
> > rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, test="t",
> > dfs="contain")
> >
> > Best,
> > Wolfgang
> >
> > > -----Original Message-----
> > > From: R-sig-meta-analysis <mailto:r-sig-meta-analysis-bounces using r-project.org>
> On
> > Behalf
> > > Of Yefeng Yang via R-sig-meta-analysis
> > > Sent: Thursday, September 26, 2024 17:35
> > > To: James Pustejovsky <mailto:jepusto using gmail.com>; R Special Interest Group
> for Meta-
> > > Analysis <mailto:r-sig-meta-analysis using r-project.org>
> > > Cc: Yefeng Yang <mailto:yefeng.yang1 using unsw.edu.au>
> > > Subject: Re: [R-meta] HKSJ adjusted error by hand
> > >
> > > Hi James,
> > >
> > > Thank you for you quick and helpful code.
> > >
> > > Interesting to see the scale factor (S_sq  in you notation) is just the
> > > heterogeneity measure H^2, which is equal to Q / df. If this is the case, I
> am
> > > wondering why this adjustment has not been extended to
> multilevel/multivariate
> > > meta-analysis, given that one can also derive the scale factor from these
> > > relative complex meta-analyses?
> > >
> > > Best,
> > > Yefeng