[R-meta] Does clubSandwich::coef_test() handle crossed random-effects?

[Farzad Keyhan]

Hi James,

Interesting! I envisioned my data structure to be as shown below. Also,
there is a bit of model fit advantage for:

random = list(~1|study / es, ~1|scale)  over  random = ~1| scale/study/es

Which would perhaps mean that true effect sizes in each row are nested in
both studies & scales and can share random-effects within and across both
studies and formulas whenever their 'study' or 'scale' ID indicators match.

Doesn't this seem more like a crossed random-effects structure?

Fred

[image: image.png]


On Sun, Oct 3, 2021 at 11:44 AM James Pustejovsky <jepusto using gmail.com> wrote:

> Ah I see. In my previous message, I think I misinterpreted what was going
> on. Based on what you've just described, it sounds like you actually do not
> have crossed random effects. Rather, you've got a hierarchical structure
> where effect sizes are nested within studies and studies are nested within
> scales. If you do not specify a clustering variable, clubSandwich defaults
> to clustering based on the highest level of random effects, so in your case
> you were clustering on the scale ID. (You could confirm this by running
> coef_test(m1, "CR2", cluster = data$scale), to see if the results are
> identical to coef_test(m1, "CR2").) If there are only a few unique values
> of the scale variable (I would guess maybe around 6 or 7?) then this would
> also explain the low degrees of freedom.
>
> So, in addition to the approach described in my previous message, a
> further alternative would be to treat scale as a set of fixed effects
> (i.e., dummy variables for each level of scale). and then cluster on
> studyID. Strictly speaking, this means that you would not be generalizing
> beyond the observed set of scales. But if you only observe 6 unique levels,
> trying to do so may be asking for more than the data can support.
>
> James
>
> On Sun, Oct 3, 2021 at 8:41 AM Farzad Keyhan <f.keyhaniha using gmail.com>
> wrote:
>
>> Hi James and Wolfgang,
>>
>> Thank you both. Please note that 'scale' doesn't vary within each
>> 'study'. Thus, to generalize beyond 'scale' levels, the only way was to add
>> it as a crossed random-effect.
>>
>> James, does that make sense to construct a V based only on 'study'
>> cluster using 'impute_covariance_matrix' but then conduct sensitivity on
>> 'r'?
>>
>> Thanks,
>> Fred
>>
>>
>> On Sun, Oct 3, 2021, 8:25 AM James Pustejovsky <jepusto using gmail.com> wrote:
>>
>>> Hi Farzad,
>>>
>>> clubSandwich does not support clustering in models with crossed random
>>> effects.
>>>
>>> There are generalizations of CRVE that work under crosses dependence
>>> structures (see Cameron, Gelbach, & Miller, 2011; Cameron & Miller, 2015).
>>> But the small-sample corrections implemented in clubSandwich have yet to be
>>> worked out for multi-way clustering. Thus, these generalizations only work
>>> when there are sufficiently large numbers of clusters *in each clustering
>>> dimension*.
>>>
>>> Based on currently available methods and software, I would suggest
>>> sticking with model-based inference (as implemented in rma.mv) if your
>>> interest is in a model with crossed random effects. That means you’d either
>>> have to do the work to develop a defensible V matrix and/or conduct
>>> sensitivity analysis for whatever assumptions go into constructing the V
>>> matrix.
>>>
>>> James
>>>
>>> Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011). Robust inference
>>> with multiway clustering. *Journal of Business & Economic Statistics*,
>>> *29*(2), 238-249.
>>>
>>> Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to
>>> cluster-robust inference. *Journal of human resources*, *50*(2),
>>> 317-372.
>>>
>>> On Oct 3, 2021, at 4:20 AM, Viechtbauer, Wolfgang (SP) <
>>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>
>>> Hi Fred,
>>>
>>> The dfs you see are computed based on a Satterthwaite approximation.
>>> Depending on the data structure, the dfs could very well be much lower than
>>> the number of studies/estimates.
>>>
>>> This aside, a limitation of the cluster robust approach is that it can
>>> only capture dependencies that occur within the groups defined by the
>>> clustering variable. Or put in a different way, for the approach to work,
>>> there need to be a sufficient (for some definition of "sufficient") number
>>> of independent groups within the dataset. When introducing crossed random
>>> effects or other random effects that can create dependencies across the
>>> entire dataset (like spatial correlation or correlation due to phylogeny in
>>> meta-analyses in ecology and evolution involing multiple species), then
>>> there might only be a single (or very few) subsets in the data that are
>>> independent and then the cluster robust approach cannot really be used
>>> anymore.
>>>
>>> Best,
>>> Wolfgang
>>>
>>> -----Original Message-----
>>>
>>> From: R-sig-meta-analysis [mailto:
>>> r-sig-meta-analysis-bounces using r-project.org] On
>>>
>>> Behalf Of Farzad Keyhan
>>>
>>> Sent: Sunday, 03 October, 2021 7:35
>>>
>>> To: R meta
>>>
>>> Subject: [R-meta] Does clubSandwich::coef_test() handle crossed
>>> random-effects?
>>>
>>>
>>> Hello Wolfgang and James,
>>>
>>>
>>> I have 51 studies, but when I run an unconditional model (below),
>>>
>>> coef_test() outputs a df of 4.76 (makes no sense).
>>>
>>>
>>> I suspected that coef_test() is not picking up my highest-level
>>>
>>> cluster (study). So, I made that explicit in the call. And then, I got
>>>
>>> an error saying "random effects are not nested".
>>>
>>>
>>> First, does clubSandwich handle models with crossed random-effects? If
>>>
>>> not, is there an alternative code for rma.mv() models to accomplish
>>>
>>> similar things (I wanted to do egger's test as well)--and of course
>>>
>>> the small sample correction may not be available there.
>>>
>>>
>>> Thanks, Fred
>>>
>>>
>>> # Note 'scale' is crossed with 'study'
>>>
>>> m1 <- rma.mv(yi, V, random = list(~1|study/es, ~1|scale), data=data)
>>>
>>>
>>> coef_test(m1, "CR2")
>>>
>>>   Coef. Estimate    SE t-stat d.f. p-val (Satt) Sig.
>>>
>>> 1 intrcpt    0.506 0.193   2.63 4.76       0.0489    *
>>>
>>>
>>> ## I made cluster explicit but get an error:
>>>
>>> coef_test(m1, "CR2", cluster = data$study)
>>>
>>> *** Error: Random effects are not nested within clustering
>>>
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211003/814a99c5/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 28256 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211003/814a99c5/attachment-0001.png>