[R-meta] Meta-analysis of within-subject experimental designs with multiple treatment factors?

[Schlüter, Elmar]

Dear Listmembers,



I am planning to conduct a meta-analysis based on data from a series of factorial survey studies. The factorial survey studies I'd like to analyze correspond to within-subject experimental designs with multiple treatment factors. However, there seem to be very few, if any, meta-analyses using data from factorial survey studies or related designs. I thus lack examples corresponding to the design I need to ask some questions about below. So here's a brief description of my data in general terms:



A factorial survey, also known as a vignette study or factorial vignette design, is a research method used to study how individuals make judgments or decisions in complex situations by presenting them with a series of hypothetical scenarios (vignettes) that vary systematically across multiple factors or attributes. For example, respondents might be asked how likely they would vote for a candidate of a political party who is



- male vs. female, (factor 1)

- has a leftwing vs. rightwing orientation (factor 2)

- and �is perceived to be very empathic/somewhat empathic/not empathic (factor 3).



This results in a 2 x 2 x 3 factorial design with 12 different vignettes.



Let�s assume that in the original data, the factor levels were measured using dummy variables. The results are unstandardised regression slopes plus standard errors. Accordingly, I'd like to meta-analyse 4 unstandardised regression slopes:



- the slope of the dummy variable for factor A (the average effect of �male� vs. �female�),

- the slope of the dummy variable for factor B (the average effect of "rightwing" vs. leftwing orientation), plus

- the two slopes of the two dummy variables for factor C (�somewhat empathic� vs �very empathic� and �not empathic� vs �very empathic�).



With one dependent variable, all studies were conducted as full or fractional factorial within-subjects designs. Therefore, I believe it would be best to account for any potential non-independence in the data, given that each respondent rated multiple (up to 12) vignettes. What I find a bit challenging here is that there is not just one average effect to examine, but four in total. Furthermore, these four average slopes result from three orthogonal factors. I (think I) have been able to combine clubSandwich and metafor to examine this data - but I would be very happy to hear if you think my approach is adequate, or if there are alternative ways to analyse this data?



Here's an illustration/sample of the data � restricted to 5 studies only, the �real� dataset will comprise of approx. 20 studies. Notice that I arranged all coefficient values in one column:





Data (df) for illustration:


study


coeff_name


coeff_value


v


coeff_index


study_1


cgenderMale


-0.07


0.000113


1


study_1


cRightwing


0.17


0.000344


2


study_1


cSomewhatEmpathic


-0.08


0.000160


3


study_1


cNotEmpathic


-0.12


0.000171


4


study_2


cgenderMale


-0.08


0.000179


5


study_2


cRightwing


0.18


0.000504


6


study_2


cSomewhatEmpathic


-0.06


0.000309


7


study_2


cNotEmpathic


-0.14


0.000435


8


study_3


cgenderMale


-0.06


0.000097


9


study_3


cRightwing


0.16


0.000319


10


study_3


cSomewhatEmpathic


-0.07


0.000150


11


study_3


cNotEmpathic


-0.11


0.000152


12


study_4


cgenderMale


-0.03


0.000107


13


study_4


cRightwing


0.15


0.000452


14


study_4


cSomewhatEmpathic


-0.05


0.000198


15


study_4


cNotEmpathic


-0.10


0.000196


16


study_5


cgenderMale


-0.04


0.000123


17


study_5


cRightwing


0.15


0.000366


18


study_5


cSomewhatEmpathic


-0.11


0.000193


19


study_5


cNotEmpathic


-0.18


0.000205


20




And here's the code I used:


library(metafor)

library(clubSandwich)

# Create sampling variance covariance matrix

V_mat <- impute_covariance_matrix(Testdata$v, cluster = df$study, r = 0.6)



# model in metafor

mod <- rma.mv(coeff_value ~ 0 + coeff_name, V = V_mat,  #coeff_name contains all coefficients as factor levels

              random = ~ 1 | study / coeff_index,   #coefficients are nested in studies

              data = df)



# clustered SEs and CIs

conf_int(mod, vcov = "CR2")



And here are the results:

Coef.


Estimate


SE


d.f.


Lower_95%CI


Upper_95%CI


coeff_namecgenderMale


-0.0553


0.00924


3.99


-0.081


-0.0296


coeff_namecRightwing


0.1623


0.0057


3.97


0.146


0.1782


coeff_namecSomewhatEmpathic


-0.0739


0.01044


3.97


-0.103


-0.0448


coeff_namecNotEmpathic


-0.1289


0.01465


3.92


-0.17


-0.0879




The code seems to works. But as I outlined at the beginning, it is not clear to if one might arrange the data in long format with all coefficients of interest in one column as I did. I thus highly appreciate your advice and any comments �



Best,

Elmar Schlueter


	[[alternative HTML version deleted]]