[R-meta] SMD from three-level nested design (raw data available)

[Fabian Schellhaas]

Dear James,

Thanks so much for your reply, this is really helpful and made me think
carefully about the data I'm dealing with. The effect I'm trying to compute
is defined by Hedges (2009, p. 348) as d_BC, i.e. the treatment effect at
level 2 of a 3-level design. In "my" dataset, the unit of measurement is
the allocation decision (level 1), and the unit of randomization is the
group (level 3). The effect I'm after, however, is the treatment effect at
the level of the participant (level 2).

Unfortunately, Hedges (2009) does not provide the equation for the
computation of d_BC using fixed-effect estimates and variance components.
However, in the context of a 2-level model, Hedges (2009) defines the
between-cluster effect as

d_B = b / sig_B  [Eq. 18.17]

where b is the estimated fixed effect and sig_B^2 is the between-cluster
variance component. Note that the within-cluster variance component is
omitted from the denominator. By contrast, the total treatment effect is
defined as

d_T = b / sqrt(sig_B^2 + sig_W^2)  [Eq. 18.23]

where b is again the estimated fixed effect, sig_B^2 is the between-cluster
variance component, and sig_W^2 is the within-cluster variance component. I
tried to apply this logic to the study I'm coding, in which the effect size
of interest is not the total treatment effect, but rather the treatment
effect at the level of individual participants (level 2). As such, I
omitted sig_w from the denominator. My understanding is that if I add the
repeated-measures variance component to the denominator, as you suggested,
I would get the treatment effect at the level of the allocation decision
(as per Hedges, 2009, Eq. 18.55). And wouldn't such an effect size be
incomparable to the other SMDs in the meta-analysis, which represent a
treatment effect at the level of participants?

Many thanks for your help,
Fabian

---
Reference:
Hedges, L. V. (2009). Effect sizes in nested designs. In Cooper, H.,
Hedges, L. V., & Valentine, J. C. (Eds.), The Handbook of Research
Synthesis and Meta-Analysis (pp. 337-355). New York: Russell Sage
Foundation.


On Sun, Nov 4, 2018 at 10:49 PM James Pustejovsky <jepusto using gmail.com> wrote:

> Fabian,
>
> Your calculations make sense to me for a two-level model (participants
> nested within groups), but you've described a three-level model. What
> happened to the other level (repeated measures, nested within
> participants)? If you have a positive variance component estimate for it,
> then I think it would make sense to include it in the denominator of the
> effect size. If X is the estimated variance of the repeated measures nested
> within participant, then take
>
> d = 6.95 / sqrt(X + 143.64 + 217.17)
>
> James
>
> On Sat, Nov 3, 2018 at 3:22 PM Fabian Schellhaas <
> fabian.schellhaas using yale.edu> wrote:
>
>> Hi all,
>>
>> I have a question about computing a standardized mean difference (SMD)
>> from
>> a primary study with a three-level nested design. The study in question
>> randomly assigned groups of participants to a treatment or control
>> condition, and then measured individual participants' resource
>> allocations.
>> While some respondents made only one such decision, others made two. As
>> such, the data in this study has three levels: resource allocation
>> decisions, which are nested in participants, which in turn are nested in
>> groups.
>>
>> I would like to compute an effect size that reflects the
>> between-participant effect of treatment vs. control. I have the raw data,
>> which the authors luckily made available. As such, I can easily fit a
>> linear mixed model with a fixed effect for treatment vs. control, and a
>> nested random effect to account for the three-level design. However, how
>> do
>> I extract a SMD from the fitted model that is comparable to SMDs from
>> single-level designs?
>>
>> The estimate for the fixed effect is 6.95, with a SE of 6.27. The variance
>> components of the random effects are 143.64 for participant nested in
>> group, and 217.17 for group. Based on formula 18.17 in Hedges (2009), I
>> believe I would compute *d* = 6.95/sqrt(143.64 + 217.17) = 0.366. However,
>> I would like to confirm that this is indeed the correct approach before I
>> proceed.
>>
>> Many thanks!
>> Fabian
>>
>> ---
>> Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology |
>> Yale
>> University
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

	[[alternative HTML version deleted]]