[R-meta] What asymmetry test to use?
Martin Lobo
m|obo4370 @end|ng |rom hotm@||@com
Thu May 2 18:36:36 CEST 2024
Hello everyone. I am performing an analysis, and when doing the asymmetry tests I see totally different results.
How do I know which one I should use?
This is a meta-analysis of SMD, with a random model given the high herogenicity.
The classic Egger test gives a p of 0.0129 and the Egger test with mixed effects model p 0.1015.
I post the results.
> regtest(fun_meta3, model = "lm")
Regression Test for Funnel Plot Asymmetry
model: weighted regression with multiplicative dispersion
predictor: standard error
test for funnel plot asymmetry: t = 2.7964, df = 16, p = 0.0129
> regtest(fun_meta3, model = "rma")
Regression Test for Funnel Plot Asymmetry
model: mixed-effects meta-regression model
predictor: standard error
test for funnel plot asymmetry: z = 1.6378, p = 0.1015
best regards
Lorenzo Martín Lobo MTSAC, FACC, FESC
Especialista Jerarquizado en Cardiología
Jefe de Dpto Enf. Cardiovasculares y Cardiometabolismo Hospital Militar Campo de Mayo.
Jefe de Cardiología Hospital Militar Campo de Mayo
Ex Jefe de Unidad Coronaria Hospital Militar Campo de Mayo
Miembro Titular de la Sociedad Argentina de Cardiología
Fellow American College of Cardiology
Fellow European Society of Cardiology
Ex Miembro del Area de Investigación de la SAC
Ex Director del Consejo de Aterosclerosis y Trombosis de la SAC
Miembro Asesor del Consejo de Aterosclerosis y Trombosis de la SAC
Ex Director del Consejo de Epidemiología y Prevención Cardiovascular de la SAC
Miembro Asesor del Consejo de Epidemiología y Prevención Cardiovascular de la SAC
Experto en Lipidos de la Sociedad Argentina de Lipidos.
Miembro de la Sociedad Argentina de Lipidos.
Instructor de ACLS de la American Heart Association
________________________________
De: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> en nombre de r-sig-meta-analysis-request using r-project.org <r-sig-meta-analysis-request using r-project.org>
Enviado: miércoles, 1 de mayo de 2024 07:00
Para: r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
Asunto: R-sig-meta-analysis Digest, Vol 84, Issue 1
Send R-sig-meta-analysis mailing list submissions to
r-sig-meta-analysis using r-project.org
To subscribe or unsubscribe via the World Wide Web, visit
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879967578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ulxX%2BrcpO0Teznp0IZzI1y%2FbHFdEUKlo8wdpCbwgpok%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
or, via email, send a message with subject or body 'help' to
r-sig-meta-analysis-request using r-project.org
You can reach the person managing the list at
r-sig-meta-analysis-owner using r-project.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-meta-analysis digest..."
Today's Topics:
1. Conceptual Question: test of moderators CHE RVE vs 3LMA
(Noah Schroeder)
2. Re: Conceptual Question: test of moderators CHE RVE vs 3LMA
(James Pustejovsky)
3. Re: Conceptual Question: test of moderators CHE RVE vs 3LMA
(Noah Schroeder)
----------------------------------------------------------------------
Message: 1
Date: Tue, 30 Apr 2024 14:07:55 -0700
From: Noah Schroeder <4noah.schroeder using gmail.com>
To: r-sig-meta-analysis using r-project.org
Subject: [R-meta] Conceptual Question: test of moderators CHE RVE vs
3LMA
Message-ID:
<CAEfFJJb9-UmWwVkJsRvkWHvsBvnSBtVvLOrzmRFdamAw-LGakQ using mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Everyone,
I am learning about CHE RVE meta-analysis and have been experimenting
with it. Something I have noticed is that occasionally, my test of
moderators pvalue will be NA. However, if I run the same code through
a typical three level structure without CHE RVE, I get a pvalue for
the test of moderators. I have the following conceptual questions I am
hoping someone may be able to help with:
1) what might cause this NA pvalue to occur in the test of moderators
when using CHE RVE?
2) Should this NA be interpreted? If so, should it be interpreted as a
non-significant result on the test of moderators? I noticed that when
this occurs i generally have notable overlap in the CIs.
3) In cases where i do get a pval for test of moderator, the levels of
the moderator with only 1 comparison do not provide tval, pval, or CI.
Is this because there is only one comparison at that level, so it is
unable to make estimates to other comparisons within that level?
For context:
3LMA code: structured similar to this code from Harrer et al.'s book:
full.model <- rma.mv(yi = z,
V = var.z,
slab = author,
data = Chernobyl,
random = ~ 1 | author/es.id,
test = "t",
method = "REML")
Conceptual overview of CHE RVE code:
set rho
set V with impute... from clubsandwich
Similar structure for 3LMA, swapping V for the set V value.
then using robust(model,... clubSandwich = TRUE).
[sorry for not providing actual code, i don't have it on this
computer. i can provide it if it is essential to answering the
question].
I am a self-taught R user, so if I am missing something really
obvious, please don't hesitate to point it out. Many, many thanks - i
have learned a lot from reading the old messages in this mailing list
(apologies if this has been addressed before and I missed it) and the
various example documentation online, and thank you all who have
developed resources for R!
Best regards,
Noah
------------------------------
Message: 2
Date: Tue, 30 Apr 2024 16:42:36 -0500
From: James Pustejovsky <jepusto using gmail.com>
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Cc: Noah Schroeder <4noah.schroeder using gmail.com>
Subject: Re: [R-meta] Conceptual Question: test of moderators CHE RVE
vs 3LMA
Message-ID:
<CAFUVuJyQdOanf7wQm2rX=_fTco20jLZRmpj7-o62SS=XaP9vtg using mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
The NAs for p-values are caused by degrees of freedom (d.f.) that are very
low, zero, or negative. In the small-sample corrections used within the
robust variance estimation framework, you can think of the d.f. as a
measure of how much information is available to estimate the standard error
of the average effect size estimate or the standard error of a beta
coefficient in a meta-regression. Very low, zero, or negative degrees of
freedom means the structure of the data is such that you can't get much (if
any) information to estimate the standard errors, so RVE is not going to
work.
This mostly occurs when one of the levels of a categorical moderator only
includes a single study or a very small number of studies. robust variance
estimators are calculated using only the _between-cluster_ variation in
outcomes. Thus, if there's only one study in a category there's no
information available to look at between-study variation within that
category; if there's only two or three studies, there's only a tiny bit of
information available to look at between-study variation---this is pretty
analogous to trying to calculate a standard deviation from a sample of only
three observations.
It's possible to get very low degrees of freedom in the RVE framework in a
situation where the model-based variance estimator (i.e., what you get from
a typical 3-level model without RVE) provides a result because the latter
uses a very different approach to estimating standard errors. For one, it
is relying on different, stronger assumptions about the structure of the
model; for another, it typically also involves some large-sample
approximation so might not be as accurate an assessment of uncertainty as
the RVE framework.
In practice, the NAs for p-values based on RVEs are really an indication
that you're using an inappropriate tool for testing the specific
hypothesis---RVE-based tests will just not give you an answer, so you
shouldn't conclude anything based on their results. One thing you can do is
reconsider the hypotheses to be tested, and perhaps limit the hypotheses to
those where you have a sufficient number of studies at each level of the
moderator. For example, maybe you start out with five categories but in the
last two of those categories there's just a single study each. You could
revise the hypothesis to test for differences only between the first three
categories because it's only there that you have a sufficient sample to
even run a test.
James
On Tue, Apr 30, 2024 at 4:08 PM Noah Schroeder via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:
> Hi Everyone,
> I am learning about CHE RVE meta-analysis and have been experimenting
> with it. Something I have noticed is that occasionally, my test of
> moderators pvalue will be NA. However, if I run the same code through
> a typical three level structure without CHE RVE, I get a pvalue for
> the test of moderators. I have the following conceptual questions I am
> hoping someone may be able to help with:
> 1) what might cause this NA pvalue to occur in the test of moderators
> when using CHE RVE?
> 2) Should this NA be interpreted? If so, should it be interpreted as a
> non-significant result on the test of moderators? I noticed that when
> this occurs i generally have notable overlap in the CIs.
> 3) In cases where i do get a pval for test of moderator, the levels of
> the moderator with only 1 comparison do not provide tval, pval, or CI.
> Is this because there is only one comparison at that level, so it is
> unable to make estimates to other comparisons within that level?
>
> For context:
> 3LMA code: structured similar to this code from Harrer et al.'s book:
>
> full.model <- rma.mv(yi = z,
> V = var.z,
> slab = author,
> data = Chernobyl,
> random = ~ 1 | author/es.id,
> test = "t",
> method = "REML")
>
>
> Conceptual overview of CHE RVE code:
> set rho
> set V with impute... from clubsandwich
> Similar structure for 3LMA, swapping V for the set V value.
> then using robust(model,... clubSandwich = TRUE).
>
> [sorry for not providing actual code, i don't have it on this
> computer. i can provide it if it is essential to answering the
> question].
>
> I am a self-taught R user, so if I am missing something really
> obvious, please don't hesitate to point it out. Many, many thanks - i
> have learned a lot from reading the old messages in this mailing list
> (apologies if this has been addressed before and I missed it) and the
> various example documentation online, and thank you all who have
> developed resources for R!
>
> Best regards,
> Noah
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879975770%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=YBoQj77C%2BaajW6357hCCrDFUbiP46k6kfb9bHhcVG%2Fw%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>
[[alternative HTML version deleted]]
------------------------------
Message: 3
Date: Tue, 30 Apr 2024 15:10:36 -0700
From: Noah Schroeder <4noah.schroeder using gmail.com>
To: James Pustejovsky <jepusto using gmail.com>
Cc: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] Conceptual Question: test of moderators CHE RVE
vs 3LMA
Message-ID:
<CAEfFJJYxboWNL0-asUiH7JJLeMfk=6uhYBk86TpmUkApUzc-0w using mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi James,
Thank you so much for the detailed reply, this helps a lot!
Have a great week everyone,
Noah
On Tue, Apr 30, 2024 at 2:42 PM James Pustejovsky <jepusto using gmail.com> wrote:
>
> The NAs for p-values are caused by degrees of freedom (d.f.) that are very low, zero, or negative. In the small-sample corrections used within the robust variance estimation framework, you can think of the d.f. as a measure of how much information is available to estimate the standard error of the average effect size estimate or the standard error of a beta coefficient in a meta-regression. Very low, zero, or negative degrees of freedom means the structure of the data is such that you can't get much (if any) information to estimate the standard errors, so RVE is not going to work.
>
> This mostly occurs when one of the levels of a categorical moderator only includes a single study or a very small number of studies. robust variance estimators are calculated using only the _between-cluster_ variation in outcomes. Thus, if there's only one study in a category there's no information available to look at between-study variation within that category; if there's only two or three studies, there's only a tiny bit of information available to look at between-study variation---this is pretty analogous to trying to calculate a standard deviation from a sample of only three observations.
>
> It's possible to get very low degrees of freedom in the RVE framework in a situation where the model-based variance estimator (i.e., what you get from a typical 3-level model without RVE) provides a result because the latter uses a very different approach to estimating standard errors. For one, it is relying on different, stronger assumptions about the structure of the model; for another, it typically also involves some large-sample approximation so might not be as accurate an assessment of uncertainty as the RVE framework.
>
> In practice, the NAs for p-values based on RVEs are really an indication that you're using an inappropriate tool for testing the specific hypothesis---RVE-based tests will just not give you an answer, so you shouldn't conclude anything based on their results. One thing you can do is reconsider the hypotheses to be tested, and perhaps limit the hypotheses to those where you have a sufficient number of studies at each level of the moderator. For example, maybe you start out with five categories but in the last two of those categories there's just a single study each. You could revise the hypothesis to test for differences only between the first three categories because it's only there that you have a sufficient sample to even run a test.
>
> James
>
> On Tue, Apr 30, 2024 at 4:08 PM Noah Schroeder via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>
>> Hi Everyone,
>> I am learning about CHE RVE meta-analysis and have been experimenting
>> with it. Something I have noticed is that occasionally, my test of
>> moderators pvalue will be NA. However, if I run the same code through
>> a typical three level structure without CHE RVE, I get a pvalue for
>> the test of moderators. I have the following conceptual questions I am
>> hoping someone may be able to help with:
>> 1) what might cause this NA pvalue to occur in the test of moderators
>> when using CHE RVE?
>> 2) Should this NA be interpreted? If so, should it be interpreted as a
>> non-significant result on the test of moderators? I noticed that when
>> this occurs i generally have notable overlap in the CIs.
>> 3) In cases where i do get a pval for test of moderator, the levels of
>> the moderator with only 1 comparison do not provide tval, pval, or CI.
>> Is this because there is only one comparison at that level, so it is
>> unable to make estimates to other comparisons within that level?
>>
>> For context:
>> 3LMA code: structured similar to this code from Harrer et al.'s book:
>>
>> full.model <- rma.mv(yi = z,
>> V = var.z,
>> slab = author,
>> data = Chernobyl,
>> random = ~ 1 | author/es.id,
>> test = "t",
>> method = "REML")
>>
>>
>> Conceptual overview of CHE RVE code:
>> set rho
>> set V with impute... from clubsandwich
>> Similar structure for 3LMA, swapping V for the set V value.
>> then using robust(model,... clubSandwich = TRUE).
>>
>> [sorry for not providing actual code, i don't have it on this
>> computer. i can provide it if it is essential to answering the
>> question].
>>
>> I am a self-taught R user, so if I am missing something really
>> obvious, please don't hesitate to point it out. Many, many thanks - i
>> have learned a lot from reading the old messages in this mailing list
>> (apologies if this has been addressed before and I missed it) and the
>> various example documentation online, and thank you all who have
>> developed resources for R!
>>
>> Best regards,
>> Noah
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879978481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=yqPs1s1Ml2PWAbaJ1%2FXYlmOnGMCEeFcb%2BqIdlz0Y1ec%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
------------------------------
Subject: Digest Footer
_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
To manage your subscription to this mailing list, go to:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C02%7C%7C65b44ea4234a4e34eb6d08dc69c5a8eb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638501544879980939%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=QEq5TgRvIrlddzbgliRZBBHBKYWXh5j73pPOIgWDYhI%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
------------------------------
End of R-sig-meta-analysis Digest, Vol 84, Issue 1
**************************************************
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list