[R-meta] Meta-analysis of within-subject experimental designs with multiple treatment factors?
James Pustejovsky
jepu@to @end|ng |rom gm@||@com
Thu Sep 14 16:22:15 CEST 2023
Hi Elmar,
The example code you provided checks out as far as I can see. Your approach
seems reasonable to me, although I also see two ways that it could be
refined:
* You've assumed a constant correlation of 0.6 between the sampling errors
of the effect size estimates from a given study. This assumption could
potentially be refined depending on how much information you have about the
study results. If you had access or could gain access to the raw data, you
could calculate better estimates of the variance-covariance matrix of these
effect estimates.
* You've used what I call the "correlated-and-hierarchical effects" working
model, which includes random effects for each individual effect and random
effects for each study. This working model assumes that 1) the overall
variation in each type of effect is constant across all four types (i.e.,
between-study heterogeneity of slope_A is the same as the between-study
heterogeneity of slope_B, likewise for slope_C1 and slope_C2) and that 2)
there is a constant correlation between the study-specific true effects of
each pair of effect types--that is, cor(slope_A, slope_B) is the same as
cor(slope_A, slope_C1) and cor(slope_A, slope_C2) and cor(slope_B,
slope_C1), etc. If you have a fair number studies, you could consider using
a more general, multivariate working model, which would loosen both of the
above assumptions. The syntax is just as above, but with random effects
specified as
random = ~ coeff_name | study, struct = "UN"
James
On Thu, Sep 14, 2023 at 4:55 AM Schlüter, Elmar via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:
> Dear Listmembers,
>
> here is a repost of my earlier mail that I unfortunately posted in
> html-format... I hope this time the data example is readable:
>
> ---
>
> I am planning to conduct a meta-analysis based on data from a series of
> factorial survey studies. The factorial survey studies I'd like to analyze
> correspond to within-subject experimental designs with multiple treatment
> factors. However, there seem to be very few, if any, meta-analyses using
> data from factorial survey studies or related designs. I thus lack examples
> corresponding to the design I need to ask some questions about below. So
> here's a brief description of my data in general terms:
>
> A factorial survey, also known as a vignette study or factorial vignette
> design, is a research method used to study how individuals make judgments
> or decisions in complex situations by presenting them with a series of
> hypothetical scenarios (vignettes) that vary systematically across multiple
> factors or attributes. For example, respondents might be asked how likely
> they would vote for a candidate of a political party who is
>
> - male vs. female, (factor 1)
>
> - has a leftwing vs. rightwing orientation (factor 2)
>
> - and behaves very empathic/somewhat empathic/not empathic (factor 3).
>
> This results in a 2 x 2 x 3 factorial design with 12 different vignettes.
>
> In the original data, the factor levels were measured using dummy
> variables. The results are unstandardised regression slopes plus standard
> errors. Accordingly, I'd like to meta-analyse 4 unstandardised regression
> slopes:
>
> - the slope of the dummy variable for factor A (the average effect of male
> vs. female), fA_slope
>
> - the slope of the dummy variable for factor B (the average effect of
> rightwing vs. leftwing orientation), fB_slope
>
> - the two slopes of the two dummy variables for factor C (somewhat
> empathic vs very empathic and not empathic vs very empathic), fC_slope1 &
> fC_slope2
>
> Using one dependent variable, all studies were conducted as full or
> fractional factorial within-subjects designs. Therefore, I believe it would
> be best to account for any potential non-independence in the data, given
> that each respondent rated multiple (up to 12) vignettes. What I find a bit
> challenging is that there is not just one average effect to examine, but
> four in total. Furthermore, these four average slopes result from three
> orthogonal factors. I (think I) have been able to combine clubSandwich and
> metafor to examine these data - but I would be very happy to hear if you
> think my approach is adequate, or if there are alternative ways to analyse
> data from such a research design?
>
> Here's an illustration/sample of the data restricted to 5 studies only,
> the real dataset will comprise of approx. 20 studies. Notice that I
> arranged all coefficient values in one column:
>
> Data (df) for illustration:
>
> study coeff_name coeff_value v coeff_index
>
> A fA_slope1 -0.07 0.000113 1
> A fB_slope1 0.17 0.000344 2
> A fC_slope1 -0.08 0.000160 3
> A fC_slope2 -0.12 0.000171 4
> B fA_slope1 -0.08 0.000179 5
> B fB_slope1 0.18 0.000504 6
> B fC_slope1 -0.06 0.000309 7
> B fC_slope2 -0.14 0.000435 8
> C fA_slope1 -0.06 0.000097 9
> C fB_slope1 0.16 0.000319 10
> C fC_slope1 -0.07 0.000150 11
> C fC_slope2 -0.11 0.000152 12
> D fA_slope1 -0.03 0.000107 13
> D fB_slope1 0.15 0.000452 14
> D fC_slope1 -0.05 0.000198 15
> D fC_slope2 -0.10 0.000196 16
> E fA_slope1 -0.04 0.000123 17
> E fB_slope1 0.15 0.000366 18
> E fC_slope1 -0.11 0.000193 19
> E fC_slope2 -0.18 0.000205 20
>
> Here's the code I used:
>
> library(metafor)
>
> library(clubSandwich)
>
> # Create sampling variance covariance matrix
>
> V_mat <- impute_covariance_matrix(Testdata$v, cluster = df$study, r = 0.6)
>
> # model in metafor
>
> mod <- rma.mv(coeff_value ~ 0 + coeff_name, V = V_mat, #coeff_name
> contains all coefficients as factor levels
>
> random = ~ 1 | study / coeff_index, #coefficients are
> nested in studies
>
> data = df)
>
> # clustered SEs and CIs
>
> conf_int(mod, vcov = "CR2")
>
> And here are the results:
>
> Coef. Estimate SE d.f.
> Lower_95%CI Upper_95%CI
> coeff_namecfA_slope1 -0.0553 0.00924 3.99 -0.081 -0.0296
> coeff_namecfB_slope1 0.1623 0.0057 3.97 0.146 0.1782
> coeff_namecfC_slope1 -0.0739 0.01044 3.97 -0.103 -0.0448
> coeff_namecfC_slope2 -0.1289 0.01465 3.92 -0.17 -0.0879
>
> The code seems to work. But as I outlined at the beginning, it is not
> clear to if one might arrange the data in "long format" with all
> coefficients of interest in one column as I did. I thus highly appreciate
> your advice and any comments
>
> Best,
> Elmar
>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list