[R-meta] Conceptual Question: test of moderators CHE RVE vs 3LMA

[Noah Schroeder]

Hi James,
 Thank you so much for the detailed reply, this helps a lot!
Have a great week everyone,
Noah

On Tue, Apr 30, 2024 at 2:42 PM James Pustejovsky <jepusto using gmail.com> wrote:
>
> The NAs for p-values are caused by degrees of freedom (d.f.) that are very low, zero, or negative. In the small-sample corrections used within the robust variance estimation framework, you can think of the d.f. as a measure of how much information is available to estimate the standard error of the average effect size estimate or the standard error of a beta coefficient in a meta-regression. Very low, zero, or negative degrees of freedom means the structure of the data is such that you can't get much (if any) information to estimate the standard errors, so RVE is not going to work.
>
> This mostly occurs when one of the levels of a categorical moderator only includes a single study or a very small number of studies. robust variance estimators are calculated using only the _between-cluster_ variation in outcomes. Thus, if there's only one study in a category there's no information available to look at between-study variation within that category; if there's only two or three studies, there's only a tiny bit of information available to look at between-study variation---this is pretty analogous to trying to calculate a standard deviation from a sample of only three observations.
>
> It's possible to get very low degrees of freedom in the RVE framework in a situation where the model-based variance estimator (i.e., what you get from a typical 3-level model without RVE) provides a result because the latter uses a very different approach to estimating standard errors. For one, it is relying on different, stronger assumptions about the structure of the model; for another, it typically also involves some large-sample approximation so might not be as accurate an assessment of uncertainty as the RVE framework.
>
> In practice, the NAs for p-values based on RVEs are really an indication that you're using an inappropriate tool for testing the specific hypothesis---RVE-based tests will just not give you an answer, so you shouldn't conclude anything based on their results. One thing you can do is reconsider the hypotheses to be tested, and perhaps limit the hypotheses to those where you have a sufficient number of studies at each level of the moderator. For example, maybe you start out with five categories but in the last two of those categories there's just a single study each. You could revise the hypothesis to test for differences only between the first three categories because it's only there that you have a sufficient sample to even run a test.
>
> James
>
> On Tue, Apr 30, 2024 at 4:08 PM Noah Schroeder via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>
>> Hi Everyone,
>>  I am learning about CHE RVE meta-analysis and have been experimenting
>> with it. Something I have noticed is that occasionally, my test of
>> moderators pvalue will be NA. However, if I run the same code through
>> a typical three level structure without CHE RVE, I get a pvalue for
>> the test of moderators. I have the following conceptual questions I am
>> hoping someone may be able to help with:
>> 1) what might cause this NA pvalue to occur in the test of moderators
>> when using CHE RVE?
>> 2) Should this NA be interpreted? If so, should it be interpreted as a
>> non-significant result on the test of moderators? I noticed that when
>> this occurs i generally have notable overlap in the CIs.
>> 3) In cases where i do get a pval for test of moderator, the levels of
>> the moderator with only 1 comparison do not provide tval, pval, or CI.
>> Is this because there is only one comparison at that level, so it is
>> unable to make estimates to other comparisons within that level?
>>
>> For context:
>> 3LMA code: structured similar to this code from Harrer et al.'s book:
>>
>> full.model <- rma.mv(yi = z,
>>                      V = var.z,
>>                      slab = author,
>>                      data = Chernobyl,
>>                      random = ~ 1 | author/es.id,
>>                      test = "t",
>>                      method = "REML")
>>
>>
>> Conceptual overview of CHE RVE code:
>> set rho
>> set V with impute... from clubsandwich
>> Similar structure for 3LMA, swapping V for the set V value.
>> then using robust(model,... clubSandwich = TRUE).
>>
>> [sorry for not providing actual code, i don't have it on this
>> computer. i can provide it if it is essential to answering the
>> question].
>>
>> I am a self-taught R user, so if I am missing something really
>> obvious, please don't hesitate to point it out. Many, many thanks - i
>> have learned a lot from reading the old messages in this mailing list
>> (apologies if this has been addressed before and I missed it) and the
>> various example documentation online, and thank you all who have
>> developed resources for R!
>>
>> Best regards,
>> Noah
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis