[R-meta] Meta-analysis of within-subject experimental designs with multiple treatment factors?

[Schlüter, Elmar]

Dear Listmembers,

here is a repost of my earlier mail that I unfortunately posted in html-format... I hope this time the data example is readable:

---

I am planning to conduct a meta-analysis based on data from a series of factorial survey studies. The factorial survey studies I'd like to analyze correspond to within-subject experimental designs with multiple treatment factors. However, there seem to be very few, if any, meta-analyses using data from factorial survey studies or related designs. I thus lack examples corresponding to the design I need to ask some questions about below. So here's a brief description of my data in general terms:

A factorial survey, also known as a vignette study or factorial vignette design, is a research method used to study how individuals make judgments or decisions in complex situations by presenting them with a series of hypothetical scenarios (vignettes) that vary systematically across multiple factors or attributes. For example, respondents might be asked how likely they would vote for a candidate of a political party who is

- male vs. female, (factor 1)

- has a leftwing vs. rightwing orientation (factor 2)

- and behaves very empathic/somewhat empathic/not empathic (factor 3).

This results in a 2 x 2 x 3 factorial design with 12 different vignettes.

In the original data, the factor levels were measured using dummy variables. The results are unstandardised regression slopes plus standard errors. Accordingly, I'd like to meta-analyse 4 unstandardised regression slopes:

- the slope of the dummy variable for factor A (the average effect of male vs. female), fA_slope

- the slope of the dummy variable for factor B (the average effect of rightwing vs. leftwing orientation), fB_slope

- the two slopes of the two dummy variables for factor C (somewhat empathic vs very empathic and not empathic vs very empathic), fC_slope1 & fC_slope2

Using one dependent variable, all studies were conducted as full or fractional factorial within-subjects designs. Therefore, I believe it would be best to account for any potential non-independence in the data, given that each respondent rated multiple (up to 12) vignettes. What I find a bit challenging is that there is not just one average effect to examine, but four in total. Furthermore, these four average slopes result from three orthogonal factors. I (think I) have been able to combine clubSandwich and metafor to examine these data - but I would be very happy to hear if you think my approach is adequate, or if there are alternative ways to analyse data from such a research design?

Here's an illustration/sample of the data   restricted to 5 studies only, the  real  dataset will comprise of approx. 20 studies. Notice that I arranged all coefficient values in one column:

Data (df) for illustration:

study coeff_name     coeff_value  v       coeff_index

A        fA_slope1      -0.07        0.000113    1
A        fB_slope1      0.17         0.000344    2
A        fC_slope1      -0.08        0.000160    3
A        fC_slope2      -0.12        0.000171    4
B        fA_slope1      -0.08        0.000179    5
B        fB_slope1      0.18         0.000504    6
B        fC_slope1      -0.06        0.000309    7
B        fC_slope2      -0.14        0.000435    8
C        fA_slope1      -0.06        0.000097    9
C        fB_slope1      0.16         0.000319    10
C        fC_slope1      -0.07        0.000150    11
C        fC_slope2      -0.11        0.000152    12
D        fA_slope1      -0.03        0.000107    13
D        fB_slope1      0.15         0.000452    14
D        fC_slope1      -0.05        0.000198    15
D        fC_slope2      -0.10        0.000196    16
E        fA_slope1      -0.04        0.000123    17
E        fB_slope1      0.15         0.000366    18
E        fC_slope1      -0.11        0.000193    19
E        fC_slope2      -0.18        0.000205    20

Here's the code I used:

library(metafor)

library(clubSandwich)

# Create sampling variance covariance matrix

V_mat <- impute_covariance_matrix(Testdata$v, cluster = df$study, r = 0.6)

# model in metafor

mod <- rma.mv(coeff_value ~ 0 + coeff_name, V = V_mat,  #coeff_name contains all coefficients as factor levels

              random = ~ 1 | study / coeff_index,   #coefficients are nested in studies

              data = df)

# clustered SEs and CIs

conf_int(mod, vcov = "CR2")

And here are the results:

Coef.                                 Estimate      SE              d.f.        Lower_95%CI Upper_95%CI
coeff_namecfA_slope1   -0.0553     0.00924     3.99     -0.081 -0.0296
coeff_namecfB_slope1   0.1623      0.0057       3.97      0.146 0.1782
coeff_namecfC_slope1   -0.0739     0.01044     3.97     -0.103 -0.0448
coeff_namecfC_slope2   -0.1289     0.01465     3.92      -0.17 -0.0879

The code seems to work. But as I outlined at the beginning, it is not clear to if one might arrange the data in "long format" with all coefficients of interest in one column as I did. I thus highly appreciate your advice and any comments

Best,
Elmar


	[[alternative HTML version deleted]]