[R-meta] SMD from three-level nested design (raw data available)

[James Pustejovsky]

I don't know of any further work on this beyond the Hedges chapter and the
two papers on which it is based. Anyone else have pointers?

On Tue, Nov 6, 2018 at 10:49 AM Fabian Schellhaas <
fabian.schellhaas using yale.edu> wrote:

> Dear James,
>
> Thanks for these clarifications, this helps a lot. Scenario 1 does indeed
> apply here, so I will add the within-subjects variance component to the
> denominator. Is there any further reading you would recommend on this topic?
>
> Many thanks,
> Fabian
>
> ---
> Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology |
> Yale University
>
>
> On Tue, Nov 6, 2018 at 11:30 AM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Fabian,
>>
>> The overarching goal in this context is to choose an effect size
>> parameter that is as comparable as possible to the other studies in the
>> synthesis. Three scenarios:
>>
>> 1. If those other studies are mostly individually randomized experiments
>> conducted across multiple contexts, but without the repeated measures
>> component, then I would argue that d_T (the average effect, standardized
>> based on the total variance of the outcome) might be more appropriate. The
>> reason is that the distribution of observed outcomes will be comprised of
>> both between-person _and within-person (between-trial)_ variation. If
>> participants respond to an instrument only once, then there is still some
>> unreliability in the resulting scores, so the corresponding variance
>> component should be included in the denominator.
>>
>> 2. If the other studies are mostly individually randomized experiments
>> conducted across narrow contexts, then it might make sense to use d_WS (eq.
>> 18.35 in Hedges, 2009), which excludes the between-group variation from the
>> denominator of the effect size. The reasoning here is that if the other
>> studies use samples that would end up as a single group in the
>> cluster-randomized trial, then the distribution of observed outcomes in
>> those studies will not include the between-group variation. For instance,
>> say that study A randomized at the school level, whereas studies B, C,
>> D,... used samples from a single school each. Then the latter studies won't
>> have between-school variation in the outcome, and we would exclude the
>> between-school component from study A in order to maintain comparability
>> with the other studies.
>>
>> 3. If the other studies mostly DID use repeated measures, but averaged
>> the scores together before analysis, then the distribution of observed
>> outcomes in those studies will not include the within-participant variation
>> (or actually it will but to a much-reduced extent). In this situation, it
>> would make sense to exclude the within-participant variance component from
>> the denominator of the effect size (and thus include only the
>> between-participant or the sum of the between-participant and between-group
>> variance components, depending on considerations analogous to the above).
>> But note that Hedges (2009) sees these effect sizes as less likely to be of
>> general interest (see notes on p. 348).
>>
>> James
>>
>> On Mon, Nov 5, 2018 at 5:31 PM Fabian Schellhaas <
>> fabian.schellhaas using yale.edu> wrote:
>>
>>> Dear James,
>>>
>>> Thanks so much for your reply, this is really helpful and made me think
>>> carefully about the data I'm dealing with. The effect I'm trying to compute
>>> is defined by Hedges (2009, p. 348) as d_BC, i.e. the treatment effect at
>>> level 2 of a 3-level design. In "my" dataset, the unit of measurement is
>>> the allocation decision (level 1), and the unit of randomization is the
>>> group (level 3). The effect I'm after, however, is the treatment effect at
>>> the level of the participant (level 2).
>>>
>>> Unfortunately, Hedges (2009) does not provide the equation for the
>>> computation of d_BC using fixed-effect estimates and variance components.
>>> However, in the context of a 2-level model, Hedges (2009) defines the
>>> between-cluster effect as
>>>
>>> d_B = b / sig_B  [Eq. 18.17]
>>>
>>> where b is the estimated fixed effect and sig_B^2 is the between-cluster
>>> variance component. Note that the within-cluster variance component is
>>> omitted from the denominator. By contrast, the total treatment effect is
>>> defined as
>>>
>>> d_T = b / sqrt(sig_B^2 + sig_W^2)  [Eq. 18.23]
>>>
>>> where b is again the estimated fixed effect, sig_B^2 is the
>>> between-cluster variance component, and sig_W^2 is the within-cluster
>>> variance component. I tried to apply this logic to the study I'm coding, in
>>> which the effect size of interest is not the total treatment effect, but
>>> rather the treatment effect at the level of individual participants (level
>>> 2). As such, I omitted sig_w from the denominator. My understanding is that
>>> if I add the repeated-measures variance component to the denominator, as
>>> you suggested, I would get the treatment effect at the level of the
>>> allocation decision (as per Hedges, 2009, Eq. 18.55). And wouldn't such an
>>> effect size be incomparable to the other SMDs in the meta-analysis, which
>>> represent a treatment effect at the level of participants?
>>>
>>> Many thanks for your help,
>>> Fabian
>>>
>>> ---
>>> Reference:
>>> Hedges, L. V. (2009). Effect sizes in nested designs. In Cooper, H.,
>>> Hedges, L. V., & Valentine, J. C. (Eds.), The Handbook of Research
>>> Synthesis and Meta-Analysis (pp. 337-355). New York: Russell Sage
>>> Foundation.
>>>
>>>
>>> On Sun, Nov 4, 2018 at 10:49 PM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>>
>>>> Fabian,
>>>>
>>>> Your calculations make sense to me for a two-level model (participants
>>>> nested within groups), but you've described a three-level model. What
>>>> happened to the other level (repeated measures, nested within
>>>> participants)? If you have a positive variance component estimate for it,
>>>> then I think it would make sense to include it in the denominator of the
>>>> effect size. If X is the estimated variance of the repeated measures nested
>>>> within participant, then take
>>>>
>>>> d = 6.95 / sqrt(X + 143.64 + 217.17)
>>>>
>>>> James
>>>>
>>>> On Sat, Nov 3, 2018 at 3:22 PM Fabian Schellhaas <
>>>> fabian.schellhaas using yale.edu> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I have a question about computing a standardized mean difference (SMD)
>>>>> from
>>>>> a primary study with a three-level nested design. The study in question
>>>>> randomly assigned groups of participants to a treatment or control
>>>>> condition, and then measured individual participants' resource
>>>>> allocations.
>>>>> While some respondents made only one such decision, others made two. As
>>>>> such, the data in this study has three levels: resource allocation
>>>>> decisions, which are nested in participants, which in turn are nested
>>>>> in
>>>>> groups.
>>>>>
>>>>> I would like to compute an effect size that reflects the
>>>>> between-participant effect of treatment vs. control. I have the raw
>>>>> data,
>>>>> which the authors luckily made available. As such, I can easily fit a
>>>>> linear mixed model with a fixed effect for treatment vs. control, and a
>>>>> nested random effect to account for the three-level design. However,
>>>>> how do
>>>>> I extract a SMD from the fitted model that is comparable to SMDs from
>>>>> single-level designs?
>>>>>
>>>>> The estimate for the fixed effect is 6.95, with a SE of 6.27. The
>>>>> variance
>>>>> components of the random effects are 143.64 for participant nested in
>>>>> group, and 217.17 for group. Based on formula 18.17 in Hedges (2009), I
>>>>> believe I would compute *d* = 6.95/sqrt(143.64 + 217.17) = 0.366.
>>>>> However,
>>>>> I would like to confirm that this is indeed the correct approach
>>>>> before I
>>>>> proceed.
>>>>>
>>>>> Many thanks!
>>>>> Fabian
>>>>>
>>>>> ---
>>>>> Fabian M. H. Schellhaas | Ph.D. Candidate | Department of Psychology |
>>>>> Yale
>>>>> University
>>>>>
>>>>>         [[alternative HTML version deleted]]
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-meta-analysis mailing list
>>>>> R-sig-meta-analysis using r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>

	[[alternative HTML version deleted]]