[R-meta] Does clubSandwich::coef_test() handle crossed random-effects?

[Farzad Keyhan]

I see, I'm still exploring to see what has caused the two models in my
previous email to give slightly different fits. Still curious though, for
'scale' and 'study' to have been crossed random effects, 'scale' should
have varied in each study?

Fred

On Sun, Oct 3, 2021 at 12:27 PM James Pustejovsky <jepusto using gmail.com> wrote:

> Fred,
>
> Your diagram does not make sense. If you are treating scale as an ID
> variable, then each unique value of scale should only be represented one
> time in the bottom row of the diagram. If you consolidate the scale IDs,
> then it looks to me that you'll end up with a hierarchical structure where
> studies are nested within scales. This structure would also be consistent
> with how you've described your data ("Please note that 'scale' doesn't
> vary within each 'study'.").
>
> Given that, I'm not sure why you would get a difference in fit between random
> = list(~1|study / es, ~1|scale) versus random = ~1| scale/study/es.
>
> James
>
>
> On Sun, Oct 3, 2021 at 12:20 PM Farzad Keyhan <f.keyhaniha using gmail.com>
> wrote:
>
>> Hi James,
>>
>> Interesting! I envisioned my data structure to be as shown below. Also,
>> there is a bit of model fit advantage for:
>>
>> random = list(~1|study / es, ~1|scale)  over  random = ~1| scale/study/es
>>
>> Which would perhaps mean that true effect sizes in each row are nested in
>> both studies & scales and can share random-effects within and across both
>> studies and formulas whenever their 'study' or 'scale' ID indicators match.
>>
>> Doesn't this seem more like a crossed random-effects structure?
>>
>> Fred
>>
>> [image: image.png]
>>
>>
>> On Sun, Oct 3, 2021 at 11:44 AM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>> Ah I see. In my previous message, I think I misinterpreted what was
>>> going on. Based on what you've just described, it sounds like you actually
>>> do not have crossed random effects. Rather, you've got a hierarchical
>>> structure where effect sizes are nested within studies and studies are
>>> nested within scales. If you do not specify a clustering variable,
>>> clubSandwich defaults to clustering based on the highest level of random
>>> effects, so in your case you were clustering on the scale ID. (You could
>>> confirm this by running
>>> coef_test(m1, "CR2", cluster = data$scale), to see if the results are
>>> identical to coef_test(m1, "CR2").) If there are only a few unique values
>>> of the scale variable (I would guess maybe around 6 or 7?) then this would
>>> also explain the low degrees of freedom.
>>>
>>> So, in addition to the approach described in my previous message, a
>>> further alternative would be to treat scale as a set of fixed effects
>>> (i.e., dummy variables for each level of scale). and then cluster on
>>> studyID. Strictly speaking, this means that you would not be generalizing
>>> beyond the observed set of scales. But if you only observe 6 unique levels,
>>> trying to do so may be asking for more than the data can support.
>>>
>>> James
>>>
>>> On Sun, Oct 3, 2021 at 8:41 AM Farzad Keyhan <f.keyhaniha using gmail.com>
>>> wrote:
>>>
>>>> Hi James and Wolfgang,
>>>>
>>>> Thank you both. Please note that 'scale' doesn't vary within each
>>>> 'study'. Thus, to generalize beyond 'scale' levels, the only way was to add
>>>> it as a crossed random-effect.
>>>>
>>>> James, does that make sense to construct a V based only on 'study'
>>>> cluster using 'impute_covariance_matrix' but then conduct sensitivity on
>>>> 'r'?
>>>>
>>>> Thanks,
>>>> Fred
>>>>
>>>>
>>>> On Sun, Oct 3, 2021, 8:25 AM James Pustejovsky <jepusto using gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Farzad,
>>>>>
>>>>> clubSandwich does not support clustering in models with crossed random
>>>>> effects.
>>>>>
>>>>> There are generalizations of CRVE that work under crosses dependence
>>>>> structures (see Cameron, Gelbach, & Miller, 2011; Cameron & Miller, 2015).
>>>>> But the small-sample corrections implemented in clubSandwich have yet to be
>>>>> worked out for multi-way clustering. Thus, these generalizations only work
>>>>> when there are sufficiently large numbers of clusters *in each clustering
>>>>> dimension*.
>>>>>
>>>>> Based on currently available methods and software, I would suggest
>>>>> sticking with model-based inference (as implemented in rma.mv) if
>>>>> your interest is in a model with crossed random effects. That means you’d
>>>>> either have to do the work to develop a defensible V matrix and/or conduct
>>>>> sensitivity analysis for whatever assumptions go into constructing the V
>>>>> matrix.
>>>>>
>>>>> James
>>>>>
>>>>> Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011). Robust
>>>>> inference with multiway clustering. *Journal of Business & Economic
>>>>> Statistics*, *29*(2), 238-249.
>>>>>
>>>>> Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to
>>>>> cluster-robust inference. *Journal of human resources*, *50*(2),
>>>>> 317-372.
>>>>>
>>>>> On Oct 3, 2021, at 4:20 AM, Viechtbauer, Wolfgang (SP) <
>>>>> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>>>>
>>>>> Hi Fred,
>>>>>
>>>>> The dfs you see are computed based on a Satterthwaite approximation.
>>>>> Depending on the data structure, the dfs could very well be much lower than
>>>>> the number of studies/estimates.
>>>>>
>>>>> This aside, a limitation of the cluster robust approach is that it can
>>>>> only capture dependencies that occur within the groups defined by the
>>>>> clustering variable. Or put in a different way, for the approach to work,
>>>>> there need to be a sufficient (for some definition of "sufficient") number
>>>>> of independent groups within the dataset. When introducing crossed random
>>>>> effects or other random effects that can create dependencies across the
>>>>> entire dataset (like spatial correlation or correlation due to phylogeny in
>>>>> meta-analyses in ecology and evolution involing multiple species), then
>>>>> there might only be a single (or very few) subsets in the data that are
>>>>> independent and then the cluster robust approach cannot really be used
>>>>> anymore.
>>>>>
>>>>> Best,
>>>>> Wolfgang
>>>>>
>>>>> -----Original Message-----
>>>>>
>>>>> From: R-sig-meta-analysis [mailto:
>>>>> r-sig-meta-analysis-bounces using r-project.org] On
>>>>>
>>>>> Behalf Of Farzad Keyhan
>>>>>
>>>>> Sent: Sunday, 03 October, 2021 7:35
>>>>>
>>>>> To: R meta
>>>>>
>>>>> Subject: [R-meta] Does clubSandwich::coef_test() handle crossed
>>>>> random-effects?
>>>>>
>>>>>
>>>>> Hello Wolfgang and James,
>>>>>
>>>>>
>>>>> I have 51 studies, but when I run an unconditional model (below),
>>>>>
>>>>> coef_test() outputs a df of 4.76 (makes no sense).
>>>>>
>>>>>
>>>>> I suspected that coef_test() is not picking up my highest-level
>>>>>
>>>>> cluster (study). So, I made that explicit in the call. And then, I got
>>>>>
>>>>> an error saying "random effects are not nested".
>>>>>
>>>>>
>>>>> First, does clubSandwich handle models with crossed random-effects? If
>>>>>
>>>>> not, is there an alternative code for rma.mv() models to accomplish
>>>>>
>>>>> similar things (I wanted to do egger's test as well)--and of course
>>>>>
>>>>> the small sample correction may not be available there.
>>>>>
>>>>>
>>>>> Thanks, Fred
>>>>>
>>>>>
>>>>> # Note 'scale' is crossed with 'study'
>>>>>
>>>>> m1 <- rma.mv(yi, V, random = list(~1|study/es, ~1|scale), data=data)
>>>>>
>>>>>
>>>>> coef_test(m1, "CR2")
>>>>>
>>>>>   Coef. Estimate    SE t-stat d.f. p-val (Satt) Sig.
>>>>>
>>>>> 1 intrcpt    0.506 0.193   2.63 4.76       0.0489    *
>>>>>
>>>>>
>>>>> ## I made cluster explicit but get an error:
>>>>>
>>>>> coef_test(m1, "CR2", cluster = data$study)
>>>>>
>>>>> *** Error: Random effects are not nested within clustering
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-meta-analysis mailing list
>>>>> R-sig-meta-analysis using r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211003/639e2c78/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 28256 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211003/639e2c78/attachment-0001.png>