[R-meta] Moderator analysis: Subsample analysis vs. model without intercept in CHE models.

[Röhl, Sebastian]

Hello,
I am currently conducting an analysis of about 500 ES in 50 teaching studies. Two ordinal moderators appear here: The teaching subject (e.g. math, language...) and the teaching aspect (e.g. athmosphere, clarity...).  I'm using a correlated and hierarchical model using "impute_covariance_matrix" from clubSandwich package.
I have a problem with differing estimates using different ways in conducting moderator analyses.
For example, I would like to analyze how ES for athmosphere differ between math and language subjects. A simple selection of the subset via "subset =" in the rma.mv function is not possible, because the imputed covariance matrix does not have the correct dimensioning.

I have compared two possibilities (syntaxes at the end of this message):
A: I select only all ES related to athmosphere in math and language subjects and impute the covariance matrix for them. Then I analyze them with the subject as moderator, using a model without intercept.
Result:
                 estimate      se�     tval�  df�    pval�    ci.lb�    ci.ub�
subject_math       0.1262  0.0204    6.1888   12   <.0001    0.0818    0.1707   ***
subject_langall   -0.0809  0.0102   -7.9658   12   <.0001   -0.1030   -0.0588   ***
(these are nearly the same estimates as using two subsets for ES with "math & atmosphere" and "language & atmosphere" and conducting separate analyses.)

B: I form two subsets for the subjects math and language, impute the covariance matrix and analyze "atmosphere" for the two subsets separately with the teaching aspects as moderator, using a model without intercept.

Math:          tq_ca           0.2088  0.0378   5.5213   17   <.0001    0.1290   0.2885   ***

Language:      tq_ca          -0.0227 0.0230 -0.9863 2.42 0.4120 -0.1065 0.0614

The numbers of effect sizes and studies are correct in the respective subsets and analyses.
Noteworthy: an analysis of all teaching aspects across all subjects in an interceptless model yields an estimate of about 0.20 for "atmosphere". In contrast, if I select only the subset with athmosphere ES, the result is an Estimate of 0.13.
Where do these large differences in the estimates come from and what would be the correct approach?

Thanks a lot for your help!

Best,
Sebastian


Syntax A:
daten_ca <- subset(daten, (daten$tq_ca==1 & (daten$subject_math==1 | daten$subject_langall==1)))
V_mat_ca <- impute_covariance_matrix(daten_ca$class_var,
                                       cluster = daten_ca$studynr,
                                       r = rho,
                                       smooth_vi = TRUE)
model_ca <- rma.mv(r_gesamt_z, V_mat_ca, random =~ 1 | studynr / nummer,
                        mods =~ -1 + subject_math + subject_langall,
                        data=daten_ca)
robust(model_ca, daten_ca$studynr)

Syntax B:
daten_math <- subset(daten, daten$subject_math==1)
daten_langall <- subset(daten, daten$subject_langall==1)
V_mat_math <- impute_covariance_matrix(daten_math$class_var,
                                       cluster = daten_math$samplenr,
                                       r = rho,
                                       smooth_vi = TRUE)
V_mat_langall <- impute_covariance_matrix(daten_langall$class_var,
                                       cluster = daten_langall$samplenr,
                                       r = rho,
                                       smooth_vi = TRUE)
model_math <- rma.mv(r_gesamt_z, V_mat_math, random =~ 1 | studynr / nummer,
                     mods =~ -1 + tq_ca + tq_cm + tq_cont + tq_pract + tq_assess +
                       tq_sup_em + tq_sup_learn + tq_adapt + tq_srl + tq_all + tq_other,
                     data=daten_math)
robust(model_math, daten_math$studynr)

model_langall <- rma.mv(r_gesamt_z, V_mat_langall, random =~ 1 | studynr / nummer,
                     mods =~ -1 + tq_ca + tq_cm + tq_cont + tq_pract + tq_assess +
                       tq_sup_em + tq_sup_learn + tq_adapt + tq_srl + tq_all + tq_other,
                     data=daten_langall)
robust(model_langall, daten_langall$studynr)

****************************
Dr. Sebastian R�hl
Eberhard Karls Universit�t T�bingen
Institute for Educational Science
T�bingen School of Education (T�SE)
Wilhelmstra�e 31 / Room 302
D-72074 T�bingen
Germany

Phone: +49 7071 29-75527
Fax: +49 7071 29-35309
Email: sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de>
Twitter: @sebastian_roehl  @ResTeacherEdu



	[[alternative HTML version deleted]]