[R-meta] Meta-analytical test of mediation model including dependent tests - looking to resolve metafor issue or find alternative approach
Lukas Wallrich
|@w@||r|ch @end|ng |rom go|d@@c@uk
Sat Dec 12 10:58:58 CET 2020
Dear Wolfgang,
Thank you very much for your quick and helpful response! The difference
indeed becomes much smaller (though it does not disappear in my case) when
I allow the heterogeneity to differ between the pairs in the joint
estimation. Now I need to decide whether that is appropriate - if I
understand Rubio-Aparicio et al. (2019) correctly, the decision depends on
whether I expect heteroscedasticity based on theory, rather than on any
test of the data? In case the data is informative, I share the full set
below - from the data and the theory, it appears that there is much more
heterogeneity in some correlations than others.
Regarding the alternative approach of combining correlation matrices: that
is actually where I started, but I did not understand how to deal with one
type of dependency: measures nested into constructs. Specifically, in my
data, some studies use two measures of the same construct, which I would
both like to use to estimate the relevant correlations. For instance,
affective and cognitive are both measures of attitudes, so they should
inform those correlations rather than be estimated differently. Is there
any way to include that into your suggested approach?
Many thanks,
Lukas
meta_data <- tibble::tribble(
~study, ~measure, ~pair, ~r, ~N, ~inv_N,
"longit", "T1", "pos_div", 0.22, 211, 0.005,
"longit", "T1", "pos_neg", 0.16, 211, 0.005,
"longit", "T1", "neg_div", -0.02, 211, 0.005,
"longit", "T2", "pos_div", 0.33, 211, 0.005,
"longit", "T2", "pos_neg", -0.05, 211, 0.005,
"longit", "T2", "neg_div", -0.28, 211, 0.005,
"UK_mediation", "only", "neg_div", -0.3, 224, 0.004,
"UK_mediation", "only", "pos_div", 0.43, 224, 0.004,
"UK_mediation", "only", "pos_neg", -0.01, 224, 0.004,
"UK_mediation", "affective", "pos_att", -0.38, 224, 0.004,
"UK_mediation", "cognitive", "pos_att", -0.2, 224, 0.004,
"UK_mediation", "affective", "div_att", -0.44, 224, 0.004,
"UK_mediation", "cognitive", "div_att", -0.55, 224, 0.004,
"UK_mediation", "affective", "neg_att", 0.18, 224, 0.004,
"UK_mediation", "cognitive", "neg_att", 0.21, 224, 0.004,
"DE_mediation", "only", "pos_div", 0.35, 2618, 0,
"DE_mediation", "only", "neg_div", -0.16, 2618, 0,
"DE_mediation", "only", "div_att", -0.53, 2618, 0,
"DE_mediation", "only", "pos_neg", 0.25, 2618, 0,
"DE_mediation", "only", "pos_att", -0.43, 2618, 0,
"DE_mediation", "only", "neg_att", 0.26, 2618, 0,
"longit", "T2_prej", "pos_att", -0.222, 211, 0.005,
"longit", "T2_prej", "neg_att", 0.137, 211, 0.005,
"longit", "T2_prej", "div_att", -0.227, 211, 0.005,
"longit", "T1_therm", "neg_att", 0.148, 211, 0.005,
"longit", "T1_therm", "div_att", -0.17, 211, 0.005,
"longit", "T1_therm", "pos_att", -0.325, 211, 0.005,
"longit", "T2_therm", "pos_att", -0.356, 211, 0.005,
"longit", "T2_therm", "neg_att", 0.103, 211, 0.005,
"longit", "T2_therm", "div_att", -0.231, 211, 0.005,
"India", "divval_pref", "pos_div", 0.14, 152, 0.007,
"India", "divval_instr", "pos_div", -0.058, 152, 0.007,
"India", "divval_pref", "neg_div", -0.016, 152, 0.007,
"India", "divval_instr", "neg_div", -0.248, 152, 0.007,
"India", "divval_pref", "pos_neg", 0.003, 152, 0.007,
"India", "divval_pref", "div_att", -0.213, 152, 0.007,
"India", "divval_instr", "div_att", -0.208, 152, 0.007,
"India", "divval_pref", "pos_att", -0.563, 152, 0.007,
"India", "divval_pref", "neg_att", -0.016, 152, 0.007,
"NCS_2018", "divval_pref", "pos_neg", -0.151, 329, 0.003,
"NCS_2018", "divval_pref", "pos_div", 0.115, 316, 0.003,
"NCS_2018", "divval_pref", "neg_div", -0.08, 315, 0.003,
"NCS_2018", "divval_better", "pos_div", 0.037, 327, 0.003,
"NCS_2018", "divval_better", "neg_div", -0.006, 326, 0.003,
"NCS_2018", "divval_pref", "pos_att", -0.264, 319, 0.003,
"NCS_2018", "divval_pref", "neg_att", 0.068, 318, 0.003,
"NCS_2018", "divval_pref", "div_att", -0.077, 317, 0.003,
"NCS_2018", "divval_better", "div_att", -0.069, 320, 0.003,
"NCS_2019", "divval_pref", "pos_neg", -0.139, 434, 0.002,
"NCS_2019", "divval_pref", "pos_div", 0.14, 110, 0.009,
"NCS_2019", "divval_pref", "neg_div", -0.167, 107, 0.009,
"NCS_2019", "divval_better", "pos_div", 0.074, 106, 0.009,
"NCS_2019", "divval_better", "neg_div", -0.206, 103, 0.01,
"NCS_2019", "divval_pref", "pos_att", -0.295, 447, 0.002,
"NCS_2019", "divval_pref", "neg_att", 0.191, 432, 0.002,
"NCS_2019", "divval_pref", "div_att", 0.126, 112, 0.009,
"NCS_2019", "divval_better", "div_att", -0.223, 107, 0.009
)
On Fri, 11 Dec 2020 at 17:32, Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> Dear Lukas,
>
> It is to be expected that the results from separate analyses will differ
> from the multilevel model. This issue, albeit in a somewhat different
> modeling context, is discussed here:
>
>
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.metafor-project.org%2Fdoku.php%2Ftips%3Acomp_two_independent_estimates&data=04%7C01%7Cl.wallrich%40gold.ac.uk%7Ca6dfed0b394b453d3c8008d89dfacbfa%7C0d431f3f20c1461c958a46b29d4e021b%7C0%7C0%7C637433047777612288%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s31zFcZ7j8Cb57iUl6THo8Wt8v9A%2BjQRbXOenPHVzTY%3D&reserved=0
>
> Also, 1/N is not quite the way the sampling variances for correlation
> coefficients should be calculated, but given that the correlations are not
> so large, this is probably not going to matter that much. One can also
> debate whether one should meta-analyze raw correlation coefficients, but
> let's leave this issue aside for now.
>
> But the results don't look strange to me. It's also a rather small
> dataset, so changes in the modeling approach can lead to noticeably
> different results.
>
> I am not sure if I would agree with the general approach here to deal with
> the multilevel structure though. Let's take the first study:
>
> 1 UK_mediation affective pos_att -0.38 0.00446
> 2 UK_mediation cognitive pos_att -0.2 0.00446
> 3 UK_mediation affective neg_att 0.18 0.00446
> 4 UK_mediation cognitive neg_att 0.21 0.00446
>
> So, as far as I can tell, there are 4 variables that were measured in this
> study: affective, cognitive, pos_att, and neg_att. If so, there should be 6
> correlations in total, but you are showing only 4 of them. If one would
> also know the affective-cognitive and the pos_att-neg_att correlations,
> then one can construct the whole 6x6 var-cov matrix of the 6 correlations
> (or their r-to-z transformed values). The 'devel' version of metafor has a
> function for this called rcalc():
>
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwviechtb.github.io%2Fmetafor%2Freference%2Frcalc.html&data=04%7C01%7Cl.wallrich%40gold.ac.uk%7Ca6dfed0b394b453d3c8008d89dfacbfa%7C0d431f3f20c1461c958a46b29d4e021b%7C0%7C0%7C637433047777612288%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Bwh12FvZuC%2B6MOSAF10Xu%2F3Pcew8zrPA%2FKXzByzp2Qc%3D&reserved=0
>
> One can then use a 'proper' multivariate model. See here for an example:
>
>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwviechtb.github.io%2Fmetafor%2Freference%2Fdat.craft2003.html&data=04%7C01%7Cl.wallrich%40gold.ac.uk%7Ca6dfed0b394b453d3c8008d89dfacbfa%7C0d431f3f20c1461c958a46b29d4e021b%7C0%7C0%7C637433047777612288%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2B4te3clH6yllgNXgeciMOM8esQ0aocOmRcuANWAnlE4%3D&reserved=0
>
> However, with 5 studies, I might even just consider using a model with a
> properly constructed V matrix and no further random effects. There doesn't
> seem to be a huge amount of heterogeneity in these data in the first place.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org]
> >On Behalf Of Lukas Wallrich
> >Sent: Friday, 11 December, 2020 17:54
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] Meta-analytical test of mediation model including
> >dependent tests - looking to resolve metafor issue or find alternative
> >approach
> >
> >Dear list members,
> >
> >I am trying to run a meta-analysis that includes dependent effect sizes
> >(correlations), nested into studies and then into measures, for which I
> >ultimately want to estimate a mediation model. For that, I am trying to
> >follow Wilson et al. (2016,
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fonlinelibrary.wiley.com%2Fdoi%2Fabs%2F10.1002%2Fjrsm.1199&data=04%7C01%7Cl.wallrich%40gold.ac.uk%7Ca6dfed0b394b453d3c8008d89dfacbfa%7C0d431f3f20c1461c958a46b29d4e021b%7C0%7C0%7C637433047777612288%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iYbqLVW2al02XYs6T6XjO9sUWKhCiJQovZnW7ChSIJc%3D&reserved=0)
> who propose to
> >collate a long dataset of each correlation value with a factor that
> >indicates which correlation it is testing, and then using that as a
> >moderator.
> >
> >However, the results differ from separate estimates of the correlation
> >coefficients and, more importantly, the confidence intervals shrink quite
> a
> >lot. I am not sure why this would happen and whether I can trust the
> >results.
> >
> >Below the code and results - I would very much appreciate any pointers
> >regarding what is going wrong, or suggestions for an alternative approach.
> >Please note that I am cross-posting this from Stackexchange where I have
> >not received a response within a week:
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstats.stackexchange.com%2Fq%2F499690%2F240420&data=04%7C01%7Cl.wallrich%40gold.ac.uk%7Ca6dfed0b394b453d3c8008d89dfacbfa%7C0d431f3f20c1461c958a46b29d4e021b%7C0%7C0%7C637433047777612288%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Dl%2Bv4pW7sH13QMmNFmDxFgEyqvaLSqKN26hJ6fG1dkU%3D&reserved=0
> >
> >Many thanks,
> >
> >Lukas
> >
> >library(metafor)
> >#> Loading required package: Matrix
> >#> Loading 'metafor' package (version 2.4-0). For an overview
> >#> and introduction to the package please type: help(metafor).
> >
> >dat <- tibble::tribble(
> > ~study, ~measure, ~pair, ~r, ~inv_N,
> > "UK_mediation", "affective", "pos_att", -0.38, 0.00446428571428571,
> > "UK_mediation", "cognitive", "pos_att", -0.2, 0.00446428571428571,
> > "UK_mediation", "affective", "neg_att", 0.18, 0.00446428571428571,
> > "UK_mediation", "cognitive", "neg_att", 0.21, 0.00446428571428571,
> > "DE_mediation", "only", "pos_att", -0.43, 0.000381970970206264,
> > "DE_mediation", "only", "neg_att", 0.26, 0.000381970970206264,
> > "longit", "T2_prej", "pos_att", -0.221742469419445, 0.004739336492891,
> > "longit", "T2_prej", "neg_att", 0.136975390214378, 0.004739336492891,
> > "longit", "T1_therm", "neg_att", 0.148356343157473, 0.004739336492891,
> > "longit", "T1_therm", "pos_att", -0.325301851215349, 0.004739336492891
> >)
> >
> >#Approach per Wilson et al. - correlation pair as moderator
> >summary(rma.mv(r,
> > inv_N,
> > random = ~ 1 | measure/study,
> > data = dat,
> > method = "REML",
> > mods = ~factor(pair)-1))
> >#>
> >#> Multivariate Meta-Analysis Model (k = 10; method: REML)
> >#>
> >#> logLik Deviance AIC BIC AICc
> >#> 2.9182 -5.8363 2.1637 2.4814 15.4970
> >#>
> >#> Variance Components:
> >#>
> >#> estim sqrt nlvls fixed factor
> >#> sigma^2.1 0.0000 0.0055 5 no measure
> >#> sigma^2.2 0.0000 0.0055 5 no measure/study
> >#>
> >#> Test for Residual Heterogeneity:
> >#> QE(df = 8) = 25.1654, p-val = 0.0015
> >#>
> >#> Test of Moderators (coefficients 1:2):
> >#> QM(df = 2) = 726.4284, p-val < .0001
> >#>
> >#> Model Results:
> >#>
> >#> estimate se zval pval ci.lb
> > ci.ub
> >#> factor(pair)neg_att 0.2389 0.0179 13.3561 <.0001 0.2038
> >0.2739 ***
> >#> factor(pair)pos_att -0.3917 0.0179 -21.8970 <.0001 -0.4267
> > -0.3566 ***
> >#>
> >#> ---
> >#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
> >
> >#Basic approach for single correlation
> >summary(rma.mv(r,
> > inv_N,
> > random = ~ 1 | measure/study,
> > data = dat[dat$pair == "pos_att",],
> > method = "REML"))
> >#>
> >#> Multivariate Meta-Analysis Model (k = 5; method: REML)
> >#>
> >#> logLik Deviance AIC BIC AICc
> >#> 3.4054 -6.8107 -0.8107 -2.6519 23.1893
> >#>
> >#> Variance Components:
> >#>
> >#> estim sqrt nlvls fixed factor
> >#> sigma^2.1 0.0040 0.0631 5 no measure
> >#> sigma^2.2 0.0040 0.0631 5 no measure/study
> >#>
> >#> Test for Heterogeneity:
> >#> Q(df = 4) = 19.1211, p-val = 0.0007
> >#>
> >#> Model Results:
> >#>
> >#> estimate se zval pval ci.lb ci.ub
> >#> -0.3224 0.0478 -6.7481 <.0001 -0.4160 -0.2287 ***
> >#>
> >#> ---
> >#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
> >
> >*Lukas Wallrich* // *Goldsmiths*, University of London
> >PhD candidate in Social Psychology
> >Department of Psychology // 216 Whitehead Building
> >New Cross // London SE14 6NW
> >l.wallrich using gold.ac.uk // +44 7591 975294
>
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list