[R-meta] rma.mv for studies reporting composite of and/or individual subscales

[Timothy MacKenzie]

>rma.mv(es ~ reporting:X1, vi, random = list(~1| study, ~ reporting |
>obs), struct = "DIAG", subset = include == "yes")

Not sure what X1 is, but yes, this could be a plausible model,
allowing for different within-study variances for 'subscale' versus
'composite' estimates.

>>>>X1 is a moderator but I think I should keep X1 separate between studies for which we have used their composite result and studies for which we have used their subscale results, no?

Tim M

On Wed, Nov 24, 2021 at 12:07 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> >-----Original Message-----
> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> >Sent: Wednesday, 24 November, 2021 17:10
> >To: Viechtbauer, Wolfgang (SP)
> >Cc: R meta
> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> >subscales
> >
> >I may have misspecified your suggested subgroup-ish model in my
> >previous email, I think the model could have been:
> >
> >rma.mv(es ~ reporting:X1, vi, random = list(~1| study, ~ reporting |
> >obs), struct = "DIAG", subset = include == "yes")
>
> Not sure what X1 is, but yes, this could be a plausible model, allowing for different within-study variances for 'subscale' versus 'composite' estimates.
>
> >Regardless, one possible downside to the subgroup model in my data is
> >that it becomes a bit subjective how to treat studies that both
> >provide separate subscales and one composite subscales results. One
> >can use only their subscales and exclude their composite part or vice
> >versa. Thus, such subjectivity may have a bearing on the results of
> >the model estimates for each subgroup depending on how one treats (c)
> >studies referenced in my first email.
>
> Sure, but excluding studies that only report a composite is also a subjective decision.
>
> >Thanks,
> >Tim M
> >
> >On Wed, Nov 24, 2021 at 9:11 AM Timothy MacKenzie <fswfswt using gmail.com> wrote:
> >>
> >> Thank you so much Wolfgang!
> >>
> >> I would tend to use (a) and (b) and for studies in group (c), I would
> >> either use an effect size computed based on the composite or the
> >> effect sizes computed based on the subscales (but not both). I would
> >> also code a moderator that indicates whether an effect size comes from
> >> a subscale or a composite measure.
> >>
> >> >>>>You mean, for example, for this data, I should only 'include' the following
> >rows?
> >>
> >> study subscale  reporting  obs  include
> >> 1        A      subscale   1             yes
> >> 1        A      subscale   2             yes
> >> 1        B      subscale   3             yes
> >> 1        B      subscale   4             yes
> >> 2        A&C    composite  5         yes
> >> 3        G&H    composite  6        yes
> >> 4        Z      subscale   7             yes
> >> 4        T      subscale   8             yes
> >> 4        Z&T    composite  9         no
> >>
> >> Then, will my model be a subgroup model like the following?
> >>
> >> rma.mv(es ~ reporting:X1, random = list(~1 | study, ~ obs |
> >> interaction(study, reporting) ), struct = "DIAG",  subset = include ==
> >> "yes")
> >>
> >> If the above model is correct, I would assume it's not meaningful to
> >> compare the fixed or random estimates for subscales with those for
> >> composites?
> >>
> >> Also, I assume I shouldn't use 'subscale' in the random part because
> >> the same subscales don't occur much across the studies, correct?
> >>
> >> Thank you very much,
> >> Tim M
> >>
> >> On Wed, Nov 24, 2021 at 7:55 AM Viechtbauer, Wolfgang (SP)
> >> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >> >
> >> > Dear Tim,
> >> >
> >> > Please see below for my responses.
> >> >
> >> > Best,
> >> > Wolfgang
> >> >
> >> > >-----Original Message-----
> >> > >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> >> > >Sent: Wednesday, 24 November, 2021 7:04
> >> > >To: R meta
> >> > >Cc: Viechtbauer, Wolfgang (SP)
> >> > >Subject: rma.mv for studies reporting composite of and/or individual
> >subscales
> >> > >
> >> > >Dear All,
> >> > >
> >> > >In my meta-analysis, I've faced two issues.
> >> > >
> >> > >First issue; each study can measure the same outcome using subscales
> >> > >reported in the following ways:
> >> > >
> >> > >(a) Some studies report only separate subscales,
> >> > >(b) Some studies report only composite of some subscales,
> >> > >(c) Some studies report both composite of and separate subscales.
> >> > >
> >> > >Second issue; the same subscales don't quite occur across different
> >> > >studies (indeed, the number of unique subscales is about the number of
> >> > >studies).
> >> > >
> >> > >To tackle the first issue, can I include only studies that report
> >> > >separate subscales from (a) and (c) studies?
> >> >
> >> > Sure you can. I don't think anybody here will come and stop you :)
> >> >
> >> > I would tend to use (a) and (b) and for studies in group (c), I would either
> >use an effect size computed based on the composite or the effect sizes computed
> >based on the subscales (but not both). For effect sizes computed based on
> >separate subscales in the same sample, the dependency between the effect sizes
> >needs to be take into consideration. I would also code a moderator that indicates
> >whether an effect size comes from a subscale or a composite measure.
> >> >
> >> > >To tackle the second issue, can I only rely on the model below (data
> >> > >structure is below)?
> >> > >
> >> > > rma.mv(es ~ 1, random = ~ 1 | study / obs, subset = subscale  ==
> >"subscale")
> >> >
> >> > I think you meant:
> >> >
> >> > rma.mv(es ~ 1, random = ~ 1 | study / obs, subset = reporting == "subscale")
> >> >
> >> > You could do that if you only want to include effect sizes computed based on
> >subscales. That would throw out studies 2 and 3. Poor studies 2 and 3 :(
> >> >
> >> > >Thank you,
> >> > >Tim M
> >> > >
> >> > >My data looks like this (please view this in a plain text editor):
> >> > >
> >> > >study subscale  reporting  obs
> >> > >1        A      subscale   1
> >> > >1        A      subscale   2
> >> > >1        B      subscale   3
> >> > >1        B      subscale   4
> >> > >2        A&C    composite  5
> >> > >3        G&H    composite  6
> >> > >4        Z      subscale   7
> >> > >4        T      subscale   8
> >> > >4        Z&T    composite  9