[R-meta] Conceptual Question: test of moderators CHE RVE vs 3LMA

[James Pustejovsky]

The NAs for p-values are caused by degrees of freedom (d.f.) that are very
low, zero, or negative. In the small-sample corrections used within the
robust variance estimation framework, you can think of the d.f. as a
measure of how much information is available to estimate the standard error
of the average effect size estimate or the standard error of a beta
coefficient in a meta-regression. Very low, zero, or negative degrees of
freedom means the structure of the data is such that you can't get much (if
any) information to estimate the standard errors, so RVE is not going to
work.

This mostly occurs when one of the levels of a categorical moderator only
includes a single study or a very small number of studies. robust variance
estimators are calculated using only the _between-cluster_ variation in
outcomes. Thus, if there's only one study in a category there's no
information available to look at between-study variation within that
category; if there's only two or three studies, there's only a tiny bit of
information available to look at between-study variation---this is pretty
analogous to trying to calculate a standard deviation from a sample of only
three observations.

It's possible to get very low degrees of freedom in the RVE framework in a
situation where the model-based variance estimator (i.e., what you get from
a typical 3-level model without RVE) provides a result because the latter
uses a very different approach to estimating standard errors. For one, it
is relying on different, stronger assumptions about the structure of the
model; for another, it typically also involves some large-sample
approximation so might not be as accurate an assessment of uncertainty as
the RVE framework.

In practice, the NAs for p-values based on RVEs are really an indication
that you're using an inappropriate tool for testing the specific
hypothesis---RVE-based tests will just not give you an answer, so you
shouldn't conclude anything based on their results. One thing you can do is
reconsider the hypotheses to be tested, and perhaps limit the hypotheses to
those where you have a sufficient number of studies at each level of the
moderator. For example, maybe you start out with five categories but in the
last two of those categories there's just a single study each. You could
revise the hypothesis to test for differences only between the first three
categories because it's only there that you have a sufficient sample to
even run a test.

James

On Tue, Apr 30, 2024 at 4:08 PM Noah Schroeder via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Hi Everyone,
>  I am learning about CHE RVE meta-analysis and have been experimenting
> with it. Something I have noticed is that occasionally, my test of
> moderators pvalue will be NA. However, if I run the same code through
> a typical three level structure without CHE RVE, I get a pvalue for
> the test of moderators. I have the following conceptual questions I am
> hoping someone may be able to help with:
> 1) what might cause this NA pvalue to occur in the test of moderators
> when using CHE RVE?
> 2) Should this NA be interpreted? If so, should it be interpreted as a
> non-significant result on the test of moderators? I noticed that when
> this occurs i generally have notable overlap in the CIs.
> 3) In cases where i do get a pval for test of moderator, the levels of
> the moderator with only 1 comparison do not provide tval, pval, or CI.
> Is this because there is only one comparison at that level, so it is
> unable to make estimates to other comparisons within that level?
>
> For context:
> 3LMA code: structured similar to this code from Harrer et al.'s book:
>
> full.model <- rma.mv(yi = z,
>                      V = var.z,
>                      slab = author,
>                      data = Chernobyl,
>                      random = ~ 1 | author/es.id,
>                      test = "t",
>                      method = "REML")
>
>
> Conceptual overview of CHE RVE code:
> set rho
> set V with impute... from clubsandwich
> Similar structure for 3LMA, swapping V for the set V value.
> then using robust(model,... clubSandwich = TRUE).
>
> [sorry for not providing actual code, i don't have it on this
> computer. i can provide it if it is essential to answering the
> question].
>
> I am a self-taught R user, so if I am missing something really
> obvious, please don't hesitate to point it out. Many, many thanks - i
> have learned a lot from reading the old messages in this mailing list
> (apologies if this has been addressed before and I missed it) and the
> various example documentation online, and thank you all who have
> developed resources for R!
>
> Best regards,
> Noah
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]