[R-meta] Potential bug in clubSandwich::Wald_test()

[Stefanou Revesz]

For sure, thank you very much.

Stefanou

On Tue, Mar 1, 2022 at 5:03 PM James Pustejovsky <jepusto using gmail.com> wrote:

> That approach uses a better variance estimator (CR2) but a very crude
> approximation to the distribution of the test statistic. I would guess that
> it would still have very poor small-sample properties for contexts such as
> your example, where the number of clusters is limited relative to the
> number of coefficients being tested.
>
> On Tue, Mar 1, 2022 at 4:00 PM Stefanou Revesz <stefanourevesz using gmail.com>
> wrote:
>
>> Thank you very much, James.  Does changing the `test=` to another option
>> (e.g., "Naive-F") but keeping the `vcov="CR2"` might still be better than
>> the QM provided by the rma.mv() model itself?
>>
>> Wald_test(fit, constrain_zero(fit$btt), "CR2", test = "Naive-F")
>>
>> Stefanou
>>
>> On Tue, Mar 1, 2022 at 2:49 PM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>> I cannot reproduce the error. Here is what I get:
>>> > Wald_test(fit, constrain_zero(fit$btt), "CR2")
>>>  test Fstat df_num df_denom p_val sig
>>>   HTZ     0     15    -0.55    NA
>>>
>>> The test you are trying to run is for a configuration where the HTZ test
>>> does not work well. It is a joint test of 15 coefficients but the data
>>> include only 28 distinct studies, so this is a very challenging scenario in
>>> terms of getting the small-sample approximation correct. In particular, the
>>> approximation leads to negative degrees of freedom and a corrected F
>>> statistic of zero because the number of coefficients is so large and the
>>> number of clusters is limited. In this situation, I would suggest using a
>>> cluster wild bootstrap test instead, as described in a recent paper by
>>> Megha Joshi and me:
>>>
>>> https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/
>>>
>>> The syntax requires only minor modifications (although it takes a bit
>>> longer to compute):
>>>
>>> > library(wildmeta)
>>> > Wald_test_cwb(fit, constraints = constrain_zero(fit$btt), R = 1999,
>>> seed = 20220301)
>>>   Test Adjustment CR_type Statistic    R      p_val
>>> 1  CWB        CR0     CR0   Naive-F 1999 0.03951976
>>>
>>> James
>>>
>>> On Tue, Mar 1, 2022 at 11:15 AM Stefanou Revesz <
>>> stefanourevesz using gmail.com> wrote:
>>>
>>>> Dear Dr. Pustejovsky (and other Meta Experts),
>>>>
>>>> I've run into a potential bug in clubSandwich::Wald_test().
>>>>
>>>> Would you please confirm whether the error in the following
>>>> reproducible example is a bug or not? Any possible fix?
>>>>
>>>> Best,
>>>> Stefanou
>>>>
>>>> d <- read.csv("https://raw.githubusercontent.com/fpqq/w/main/f.csv")
>>>>
>>>> fit <- rma.mv(gi ~ teaching_level +pd_format +pd_content +
>>>>                       training_hr +time +outcome +area +
>>>>                        time_wk_whn +time_wk_btw +study_type,
>>>>              V = v_gi,
>>>>              random = ~1|study/obs,
>>>>              data = d)
>>>>
>>>> Wald_test(fit, constrain_zero(fit$btt), "CR2")
>>>>
>>>> Error in symnum(x$p_val, corr = FALSE, na = FALSE, cutpoints = c(0,
>>>> 0.001,  :
>>>>   must have 2 'symbols' for logical 'x' argument
>>>>
>>>

	[[alternative HTML version deleted]]