[R-meta] SMD from three-level nested design (raw data available)

[Fabian Schellhaas]

Dear James,

Thanks for these clarifications, this helps a lot. Scenario 1 does indeed
apply here, so I will add the within-subjects variance component to the
denominator. Is there any further reading you would recommend on this topic?

Many thanks,
Fabian

---
Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology | Yale
University


On Tue, Nov 6, 2018 at 11:30 AM James Pustejovsky <jepusto using gmail.com> wrote:

> Fabian,
>
> The overarching goal in this context is to choose an effect size parameter
> that is as comparable as possible to the other studies in the synthesis.
> Three scenarios:
>
> 1. If those other studies are mostly individually randomized experiments
> conducted across multiple contexts, but without the repeated measures
> component, then I would argue that d_T (the average effect, standardized
> based on the total variance of the outcome) might be more appropriate. The
> reason is that the distribution of observed outcomes will be comprised of
> both between-person _and within-person (between-trial)_ variation. If
> participants respond to an instrument only once, then there is still some
> unreliability in the resulting scores, so the corresponding variance
> component should be included in the denominator.
>
> 2. If the other studies are mostly individually randomized experiments
> conducted across narrow contexts, then it might make sense to use d_WS (eq.
> 18.35 in Hedges, 2009), which excludes the between-group variation from the
> denominator of the effect size. The reasoning here is that if the other
> studies use samples that would end up as a single group in the
> cluster-randomized trial, then the distribution of observed outcomes in
> those studies will not include the between-group variation. For instance,
> say that study A randomized at the school level, whereas studies B, C,
> D,... used samples from a single school each. Then the latter studies won't
> have between-school variation in the outcome, and we would exclude the
> between-school component from study A in order to maintain comparability
> with the other studies.
>
> 3. If the other studies mostly DID use repeated measures, but averaged the
> scores together before analysis, then the distribution of observed outcomes
> in those studies will not include the within-participant variation (or
> actually it will but to a much-reduced extent). In this situation, it would
> make sense to exclude the within-participant variance component from the
> denominator of the effect size (and thus include only the
> between-participant or the sum of the between-participant and between-group
> variance components, depending on considerations analogous to the above).
> But note that Hedges (2009) sees these effect sizes as less likely to be of
> general interest (see notes on p. 348).
>
> James
>
> On Mon, Nov 5, 2018 at 5:31 PM Fabian Schellhaas <
> fabian.schellhaas using yale.edu> wrote:
>
>> Dear James,
>>
>> Thanks so much for your reply, this is really helpful and made me think
>> carefully about the data I'm dealing with. The effect I'm trying to compute
>> is defined by Hedges (2009, p. 348) as d_BC, i.e. the treatment effect at
>> level 2 of a 3-level design. In "my" dataset, the unit of measurement is
>> the allocation decision (level 1), and the unit of randomization is the
>> group (level 3). The effect I'm after, however, is the treatment effect at
>> the level of the participant (level 2).
>>
>> Unfortunately, Hedges (2009) does not provide the equation for the
>> computation of d_BC using fixed-effect estimates and variance components.
>> However, in the context of a 2-level model, Hedges (2009) defines the
>> between-cluster effect as
>>
>> d_B = b / sig_B  [Eq. 18.17]
>>
>> where b is the estimated fixed effect and sig_B^2 is the between-cluster
>> variance component. Note that the within-cluster variance component is
>> omitted from the denominator. By contrast, the total treatment effect is
>> defined as
>>
>> d_T = b / sqrt(sig_B^2 + sig_W^2)  [Eq. 18.23]
>>
>> where b is again the estimated fixed effect, sig_B^2 is the
>> between-cluster variance component, and sig_W^2 is the within-cluster
>> variance component. I tried to apply this logic to the study I'm coding, in
>> which the effect size of interest is not the total treatment effect, but
>> rather the treatment effect at the level of individual participants (level
>> 2). As such, I omitted sig_w from the denominator. My understanding is that
>> if I add the repeated-measures variance component to the denominator, as
>> you suggested, I would get the treatment effect at the level of the
>> allocation decision (as per Hedges, 2009, Eq. 18.55). And wouldn't such an
>> effect size be incomparable to the other SMDs in the meta-analysis, which
>> represent a treatment effect at the level of participants?
>>
>> Many thanks for your help,
>> Fabian
>>
>> ---
>> Reference:
>> Hedges, L. V. (2009). Effect sizes in nested designs. In Cooper, H.,
>> Hedges, L. V., & Valentine, J. C. (Eds.), The Handbook of Research
>> Synthesis and Meta-Analysis (pp. 337-355). New York: Russell Sage
>> Foundation.
>>
>>
>> On Sun, Nov 4, 2018 at 10:49 PM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>> Fabian,
>>>
>>> Your calculations make sense to me for a two-level model (participants
>>> nested within groups), but you've described a three-level model. What
>>> happened to the other level (repeated measures, nested within
>>> participants)? If you have a positive variance component estimate for it,
>>> then I think it would make sense to include it in the denominator of the
>>> effect size. If X is the estimated variance of the repeated measures nested
>>> within participant, then take
>>>
>>> d = 6.95 / sqrt(X + 143.64 + 217.17)
>>>
>>> James
>>>
>>> On Sat, Nov 3, 2018 at 3:22 PM Fabian Schellhaas <
>>> fabian.schellhaas using yale.edu> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have a question about computing a standardized mean difference (SMD)
>>>> from
>>>> a primary study with a three-level nested design. The study in question
>>>> randomly assigned groups of participants to a treatment or control
>>>> condition, and then measured individual participants' resource
>>>> allocations.
>>>> While some respondents made only one such decision, others made two. As
>>>> such, the data in this study has three levels: resource allocation
>>>> decisions, which are nested in participants, which in turn are nested in
>>>> groups.
>>>>
>>>> I would like to compute an effect size that reflects the
>>>> between-participant effect of treatment vs. control. I have the raw
>>>> data,
>>>> which the authors luckily made available. As such, I can easily fit a
>>>> linear mixed model with a fixed effect for treatment vs. control, and a
>>>> nested random effect to account for the three-level design. However,
>>>> how do
>>>> I extract a SMD from the fitted model that is comparable to SMDs from
>>>> single-level designs?
>>>>
>>>> The estimate for the fixed effect is 6.95, with a SE of 6.27. The
>>>> variance
>>>> components of the random effects are 143.64 for participant nested in
>>>> group, and 217.17 for group. Based on formula 18.17 in Hedges (2009), I
>>>> believe I would compute *d* = 6.95/sqrt(143.64 + 217.17) = 0.366.
>>>> However,
>>>> I would like to confirm that this is indeed the correct approach before
>>>> I
>>>> proceed.
>>>>
>>>> Many thanks!
>>>> Fabian
>>>>
>>>> ---
>>>> Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology |
>>>> Yale
>>>> University
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> R-sig-meta-analysis mailing list
>>>> R-sig-meta-analysis using r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>
>>>

	[[alternative HTML version deleted]]