[R-meta] Meta-analysis of within-subject experimental designs with multiple treatment factors?

Schlüter, Elmar E|m@r@Sch|ueter @end|ng |rom @ow|@un|-g|e@@en@de
Fri Sep 15 09:12:45 CEST 2023 

Dear James,


I would just like to say "thank you" for your quick and helpful reply - I will be happy to try and adapt my models accordingly.


Best,

Elmar

________________________________
Von: James Pustejovsky <jepusto using gmail.com>
Gesendet: Donnerstag, 14. September 2023 16:22:15
An: R Special Interest Group for Meta-Analysis
Cc: Schlüter, Elmar
Betreff: Re: [R-meta] Meta-analysis of within-subject experimental designs with multiple treatment factors?

Hi Elmar,

The example code you provided checks out as far as I can see. Your approach seems reasonable to me, although I also see two ways that it could be refined:

* You've assumed a constant correlation of 0.6 between the sampling errors of the effect size estimates from a given study. This assumption could potentially be refined depending on how much information you have about the study results. If you had access or could gain access to the raw data, you could calculate better estimates of the variance-covariance matrix of these effect estimates.

* You've used what I call the "correlated-and-hierarchical effects" working model, which includes random effects for each individual effect and random effects for each study. This working model assumes that 1) the overall variation in each type of effect is constant across all four types (i.e., between-study heterogeneity of slope_A is the same as the between-study heterogeneity of slope_B, likewise for slope_C1 and slope_C2) and that 2) there is a constant correlation between the study-specific true effects of each pair of effect types--that is, cor(slope_A, slope_B) is the same as cor(slope_A, slope_C1) and cor(slope_A, slope_C2) and cor(slope_B, slope_C1), etc. If you have a fair number studies, you could consider using a more general, multivariate working model, which would loosen both of the above assumptions. The syntax is just as above, but with random effects specified as
random = ~ coeff_name | study, struct = "UN"

James

On Thu, Sep 14, 2023 at 4:55 AM Schlüter, Elmar via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>> wrote:
Dear Listmembers,

here is a repost of my earlier mail that I unfortunately posted in html-format... I hope this time the data example is readable:

---

I am planning to conduct a meta-analysis based on data from a series of factorial survey studies. The factorial survey studies I'd like to analyze correspond to within-subject experimental designs with multiple treatment factors. However, there seem to be very few, if any, meta-analyses using data from factorial survey studies or related designs. I thus lack examples corresponding to the design I need to ask some questions about below. So here's a brief description of my data in general terms:

A factorial survey, also known as a vignette study or factorial vignette design, is a research method used to study how individuals make judgments or decisions in complex situations by presenting them with a series of hypothetical scenarios (vignettes) that vary systematically across multiple factors or attributes. For example, respondents might be asked how likely they would vote for a candidate of a political party who is

- male vs. female, (factor 1)

- has a leftwing vs. rightwing orientation (factor 2)

- and behaves very empathic/somewhat empathic/not empathic (factor 3).

This results in a 2 x 2 x 3 factorial design with 12 different vignettes.

In the original data, the factor levels were measured using dummy variables. The results are unstandardised regression slopes plus standard errors. Accordingly, I'd like to meta-analyse 4 unstandardised regression slopes:

- the slope of the dummy variable for factor A (the average effect of male vs. female), fA_slope

- the slope of the dummy variable for factor B (the average effect of rightwing vs. leftwing orientation), fB_slope

- the two slopes of the two dummy variables for factor C (somewhat empathic vs very empathic and not empathic vs very empathic), fC_slope1 & fC_slope2

Using one dependent variable, all studies were conducted as full or fractional factorial within-subjects designs. Therefore, I believe it would be best to account for any potential non-independence in the data, given that each respondent rated multiple (up to 12) vignettes. What I find a bit challenging is that there is not just one average effect to examine, but four in total. Furthermore, these four average slopes result from three orthogonal factors. I (think I) have been able to combine clubSandwich and metafor to examine these data - but I would be very happy to hear if you think my approach is adequate, or if there are alternative ways to analyse data from such a research design?

Here's an illustration/sample of the data   restricted to 5 studies only, the  real  dataset will comprise of approx. 20 studies. Notice that I arranged all coefficient values in one column:

Data (df) for illustration:

study coeff_name     coeff_value  v       coeff_index

A        fA_slope1      -0.07        0.000113    1
A        fB_slope1      0.17         0.000344    2
A        fC_slope1      -0.08        0.000160    3
A        fC_slope2      -0.12        0.000171    4
B        fA_slope1      -0.08        0.000179    5
B        fB_slope1      0.18         0.000504    6
B        fC_slope1      -0.06        0.000309    7
B        fC_slope2      -0.14        0.000435    8
C        fA_slope1      -0.06        0.000097    9
C        fB_slope1      0.16         0.000319    10
C        fC_slope1      -0.07        0.000150    11
C        fC_slope2      -0.11        0.000152    12
D        fA_slope1      -0.03        0.000107    13
D        fB_slope1      0.15         0.000452    14
D        fC_slope1      -0.05        0.000198    15
D        fC_slope2      -0.10        0.000196    16
E        fA_slope1      -0.04        0.000123    17
E        fB_slope1      0.15         0.000366    18
E        fC_slope1      -0.11        0.000193    19
E        fC_slope2      -0.18        0.000205    20

Here's the code I used:

library(metafor)

library(clubSandwich)

# Create sampling variance covariance matrix

V_mat <- impute_covariance_matrix(Testdata$v, cluster = df$study, r = 0.6)

# model in metafor

mod <- rma.mv<http://rma.mv>(coeff_value ~ 0 + coeff_name, V = V_mat,  #coeff_name contains all coefficients as factor levels

              random = ~ 1 | study / coeff_index,   #coefficients are nested in studies

              data = df)

# clustered SEs and CIs

conf_int(mod, vcov = "CR2")

And here are the results:

Coef.                                 Estimate      SE              d.f.        Lower_95%CI Upper_95%CI
coeff_namecfA_slope1   -0.0553     0.00924     3.99     -0.081 -0.0296
coeff_namecfB_slope1   0.1623      0.0057       3.97      0.146 0.1782
coeff_namecfC_slope1   -0.0739     0.01044     3.97     -0.103 -0.0448
coeff_namecfC_slope2   -0.1289     0.01465     3.92      -0.17 -0.0879

The code seems to work. But as I outlined at the beginning, it is not clear to if one might arrange the data in "long format" with all coefficients of interest in one column as I did. I thus highly appreciate your advice and any comments

Best,
Elmar


        [[alternative HTML version deleted]]

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org<mailto:R-sig-meta-analysis using r-project.org>
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

	[[alternative HTML version deleted]]

More information about the R-sig-meta-analysis mailing list