[R-meta] cluster-randomized trials

[Elli J. Theobald]

 Greetings -

I am working on a meta-analysis of studies that measured student
achievement in university classrooms. Most often, the treatment (and
control) are randomized to groups of students (e.g., two sections of the
same course, one section receives the treatment and one section receives
the control). My understanding from Hedges 2007 (as well as from my own
understanding of multilevel modeling) is that this type of cluster-(or
group-)randomization can greatly influence the accuracy of the estimate of
the treatment effect because students here are not truly independent.

The Konstantopoulos 2011 paper, which is beautifully detailed in the
Analysis Examples on the metafor website, is very similar to the issue I am
describing but can account for this clustering with a school random effect
(although, it is arguable that they should also include a classroom random
effect). In my case, it is not possible to add a 'section' effect because
section is synonymous with treatment (and because the effect sizes are
being calculated as differences between sections).

I also understand that correcting for this clustering is possible if the
intraclass correlation is known. In my case, is easily approximated from
the general literature (e.g., Hedges and Hedberg 2007), previous
meta-analyses (e.g., Freeman et al. 2014), or computed from data from our
own institution.

I am wondering two things:
1) is there a way to account for the clustering inherent in
cluster-randomized trials that is already built into metafor? It is
entirely possible that I missed it when I scoured the package! It would be
ideal to be able to show results from several values of rho (intraclass
correlation) to see how robust the estimates are to various values.

2) what are your thoughts about this correction? The corrections that I
have seen (e.g., in Freeman et al. 2014) are very harsh; for example, with
rho=0.22 (as used in Freeman et al. 2014 and cited from elementary classes
in Hedges and Hedberg 2007) effective sample size drops from a couple
hundred to only nine! Is this necessary?

There are a couple of additional complicating matters about the data (data
are never "neat" but I almost always find myself wishing they were!):

First, there are several studies that report means from one control section
and several treatment sections (for example, StudyID 113 below). I feel
confident dealing with this, given the great description of Gleser and
Olkin 2009 on the metafor website, about multiple treatment studies.
However, at what point should the cluster-correction be applied? Intuition
tells me after calculating the effect size and before calculating the
variance-covariance matrix. Do you agree?

Second, there are several studies that report results from several control
sections and several treatment sections, with no obvious pairings. In other
words, there is no reason that the effect size should be calculated between
treatment section 1 and control section 1 (and not control section 2, 3, 4,
etc.; e.g. StudyID 2 below). This is complicated because each of these
sections suffers from this cluster-randomized quirk so needs to be
accounted for. My original plan was to try this analysis two ways: first by
averaging (weighted) all the control sections and calculating effect sizes
for each of the treatments then applying the cluster correction. Then to
confirm that the results are comparable, compare the final model(s) from
this to final models wherein I first average all the treatment sections and
calculate effect sizes for each of the control comparisons, then applying
the cluster correction. This, however, does not actually account for the
clustering at the right level. There is clustering in each of the sections
that gets averaged. Is there anything better that I can do?

And finally, a second analysis that I am planning (on a subset of the
papers that report data in the necessary way; these data are not shown
below) is to look at sub-groups of students in these sections and calculate
performance differences between these sub-groups. For example, asking if
men and women in the control and treatment sections perform differently; in
other words, does the treatment have a disproportionate effect on a
particular sub-group of students. The trouble here is that treatment is
still cluster-randomized but student sub-groups should be truly randomized
(at least most of the time) to sections. Would it be appropriate to apply
the cluster correction to the weighted achievement difference between the
two groups?

Below is a subset of the data I am working with which illustrates the
issues above. I would be happy to send these data as R-readable if that is
desirable. I could also provide an example of the analysis of student
sub-groups if that is desirable.

Thank you for any guidance you can give!

Cheers.
Elli


Example data:
StudyID = unique study identifier
CMean = mean from control section
CSD = control section standard deviation
Cn = number of students in control section
TrtMean = mean from treatment section
TrtSD = treatment section standard deviation
Trtn = number of students in treatment section
Ni = number of students total in the study.

StudyID -- CMean -- CSD -- Cn -- TrtMean -- TrtSD -- Trtn -- Ni
1                0.20         0.39    67      0.19          0.41       73
    140
2                3.12         1.35    534    3.36          1.05       47
  581
2                3.08         1.37    534    3.38          0.87       47
  581
2                2.60         1.45    534    2.70          1.08       47
  581
2                2.40         1.78    534    2.23          1.63       47
  581
2                2.37         1.63    534    2.49          1.47       47
  581
3                44.46       20.9    154    47.30        20.03     352
506
4                34.40       8.68     60     35.45        8.51       62
  122
113            58.10       24.40   49     63.20        21.80     62
 294
113            58.10       24.40   49     64.00        21.60     85
 294
113            58.10       24.40   49     64.10        20.80     98
 294



*References*:
Freeman, S., Eddy, S.L., McDonough, M., Smith, M.K., Okoroafor, N., Jordt,
H. and Wenderoth, M.P., 2014. Active learning increases student performance
in science, engineering, and mathematics. *Proceedings of the National
Academy of Sciences*, *111*(23), pp.8410-8415.

Hedges, L.V., 2007. Correcting a significance test for clustering.

Hedges, L.V. and Hedberg, E.C., 2007. Intraclass correlation values for
planning group-randomized trials in education. *Educational Evaluation and
Policy Analysis*, *29*(1), pp.60-87.



-- 
<http://www.biology.washington.edu/users/elli-jenkins>Elli J. Theobald, PhD
Postdoctoral Research Associate
Biology Education Research Group
Department of Biology
University of Washington, Seattle
https://sites.google.com/site/ellijtheobald
<https://sites.google.com/site/ellijtheobald/home>

	[[alternative HTML version deleted]]