[R-meta] Does clubSandwich::coef_test() handle crossed random-effects?

[James Pustejovsky]

Ah I see. In my previous message, I think I misinterpreted what was going
on. Based on what you've just described, it sounds like you actually do not
have crossed random effects. Rather, you've got a hierarchical structure
where effect sizes are nested within studies and studies are nested within
scales. If you do not specify a clustering variable, clubSandwich defaults
to clustering based on the highest level of random effects, so in your case
you were clustering on the scale ID. (You could confirm this by running
coef_test(m1, "CR2", cluster = data$scale), to see if the results are
identical to coef_test(m1, "CR2").) If there are only a few unique values
of the scale variable (I would guess maybe around 6 or 7?) then this would
also explain the low degrees of freedom.

So, in addition to the approach described in my previous message, a further
alternative would be to treat scale as a set of fixed effects (i.e., dummy
variables for each level of scale). and then cluster on studyID. Strictly
speaking, this means that you would not be generalizing beyond the observed
set of scales. But if you only observe 6 unique levels, trying to do so may
be asking for more than the data can support.

James

On Sun, Oct 3, 2021 at 8:41 AM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:

> Hi James and Wolfgang,
>
> Thank you both. Please note that 'scale' doesn't vary within each 'study'.
> Thus, to generalize beyond 'scale' levels, the only way was to add it as a
> crossed random-effect.
>
> James, does that make sense to construct a V based only on 'study' cluster
> using 'impute_covariance_matrix' but then conduct sensitivity on 'r'?
>
> Thanks,
> Fred
>
>
> On Sun, Oct 3, 2021, 8:25 AM James Pustejovsky <jepusto using gmail.com> wrote:
>
>> Hi Farzad,
>>
>> clubSandwich does not support clustering in models with crossed random
>> effects.
>>
>> There are generalizations of CRVE that work under crosses dependence
>> structures (see Cameron, Gelbach, & Miller, 2011; Cameron & Miller, 2015).
>> But the small-sample corrections implemented in clubSandwich have yet to be
>> worked out for multi-way clustering. Thus, these generalizations only work
>> when there are sufficiently large numbers of clusters *in each clustering
>> dimension*.
>>
>> Based on currently available methods and software, I would suggest
>> sticking with model-based inference (as implemented in rma.mv) if your
>> interest is in a model with crossed random effects. That means you’d either
>> have to do the work to develop a defensible V matrix and/or conduct
>> sensitivity analysis for whatever assumptions go into constructing the V
>> matrix.
>>
>> James
>>
>> Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011). Robust inference
>> with multiway clustering. *Journal of Business & Economic Statistics*,
>> *29*(2), 238-249.
>>
>> Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to
>> cluster-robust inference. *Journal of human resources*, *50*(2), 317-372.
>>
>> On Oct 3, 2021, at 4:20 AM, Viechtbauer, Wolfgang (SP) <
>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> Hi Fred,
>>
>> The dfs you see are computed based on a Satterthwaite approximation.
>> Depending on the data structure, the dfs could very well be much lower than
>> the number of studies/estimates.
>>
>> This aside, a limitation of the cluster robust approach is that it can
>> only capture dependencies that occur within the groups defined by the
>> clustering variable. Or put in a different way, for the approach to work,
>> there need to be a sufficient (for some definition of "sufficient") number
>> of independent groups within the dataset. When introducing crossed random
>> effects or other random effects that can create dependencies across the
>> entire dataset (like spatial correlation or correlation due to phylogeny in
>> meta-analyses in ecology and evolution involing multiple species), then
>> there might only be a single (or very few) subsets in the data that are
>> independent and then the cluster robust approach cannot really be used
>> anymore.
>>
>> Best,
>> Wolfgang
>>
>> -----Original Message-----
>>
>> From: R-sig-meta-analysis [mailto:
>> r-sig-meta-analysis-bounces using r-project.org] On
>>
>> Behalf Of Farzad Keyhan
>>
>> Sent: Sunday, 03 October, 2021 7:35
>>
>> To: R meta
>>
>> Subject: [R-meta] Does clubSandwich::coef_test() handle crossed
>> random-effects?
>>
>>
>> Hello Wolfgang and James,
>>
>>
>> I have 51 studies, but when I run an unconditional model (below),
>>
>> coef_test() outputs a df of 4.76 (makes no sense).
>>
>>
>> I suspected that coef_test() is not picking up my highest-level
>>
>> cluster (study). So, I made that explicit in the call. And then, I got
>>
>> an error saying "random effects are not nested".
>>
>>
>> First, does clubSandwich handle models with crossed random-effects? If
>>
>> not, is there an alternative code for rma.mv() models to accomplish
>>
>> similar things (I wanted to do egger's test as well)--and of course
>>
>> the small sample correction may not be available there.
>>
>>
>> Thanks, Fred
>>
>>
>> # Note 'scale' is crossed with 'study'
>>
>> m1 <- rma.mv(yi, V, random = list(~1|study/es, ~1|scale), data=data)
>>
>>
>> coef_test(m1, "CR2")
>>
>>   Coef. Estimate    SE t-stat d.f. p-val (Satt) Sig.
>>
>> 1 intrcpt    0.506 0.193   2.63 4.76       0.0489    *
>>
>>
>> ## I made cluster explicit but get an error:
>>
>> coef_test(m1, "CR2", cluster = data$study)
>>
>> *** Error: Random effects are not nested within clustering
>>
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>>

	[[alternative HTML version deleted]]