[R-meta] rma.mv for studies reporting composite of and/or individual subscales

[Timothy MacKenzie]

By standard view, I mean the usual/standard way in which construct X
should be measured (i.e., using a scale whose subscales are A, B, C).

Under this standard view, when we look at the literature, we see:

1- Some studies mixed and matched the usual subscales to create their
unique composites (e.g., AB;  AC;  ABC)
2- Some studies report some or all these usual subscales and report
each separately (e.g., A,B;  A,C;  A,B,C)

By alternative view, I mean the researcher-constructed ways in which
construct X can be measured (i.e., using ANY scale whose subscales can
be ANYTHING appropriate to the researchers e.g., E,F,G ...).

Under this alternative view, when we look at the literature, we see:

(3) Some studies mixed and matched their own subscales to create their
unique composites (e.g., EF;  EG;  EFG),
(4) Some studies report some or all such subscales separately (e.g.,
E,F;  E,G;  E,F,G)

The result of such a trend is the data structure below. Therefore,
there are three gray areas for me:

1) Dealing with composite vs separate subscales (resolved:-)
2) Dealing with whether effects have been obtained under standard or
alternative view (maybe this should be a moderator?)
3) How should this data be subgrouped i.e., only by composite vs.
subscales or by standard vs. alternative view?

Thanks, Tim M

study subscale  reporting  obs  include
1        A      subscale   1    yes
1        A      subscale   2    yes
1        B      subscale   3    yes
1        B      subscale   4    yes
2        A&C    composite  5    yes
3        G&F    composite  6    yes
4        E      subscale   7    yes
4        F      subscale   8    yes
4        E&F    composite  9    no

On Wed, Nov 24, 2021 at 2:29 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Sorry, I can't follow. What is 'standard view' and 'alternative view'? Those sound the same to me, except for different letters.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> >Sent: Wednesday, 24 November, 2021 20:56
> >To: Viechtbauer, Wolfgang (SP)
> >Cc: R meta
> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> >subscales
> >
> >Appreciate it. Thank you very much. My response is below inline.
> >
> >Say there are also some studies that, for some reason, have broken
> >down such a scale into a few subscales, say BDI1 and BDI2, and they do
> >not report means and SDs for the overall BDI scale, only for these
> >subscales.
> >
> >BDI is a mixture of BDI1 and BDI2 anyway, so if I only have BDI, then
> >this is what the effect size reflects. If I include effect sizes based
> >on BDI1 and BDI2 in the analysis, then the model essentially mixes
> >them together.
> >
> >>>>>Sure, but, what if
> >**one the one hand**: BDI *in standard view* is a mixture of A, B and
> >C subscales and (1) some studies can mix and match them to create
> >their unique composites (AB;  AC;  ABC), (2) some studies report some
> >or all these subscales (A,B;  A,C;  A,B,C), and
> >
> >**on the other hand**: BDI *in alternative view* is a mixture of E, F
> >and G subscales and (3) some studies can mix and match them to create
> >their unique composites (EF;  EG;  EFG), and (4) some studies report
> >some or all these subscales (E,F;  E,G;  E,F,G)?
> >
> >This is what is reflected in my data structure below (as mentioned
> >earlier, the number of unique subscales is about the number of
> >studies).
> >
> >Thanks, Tim M
> >
> >study subscale  reporting  obs  include
> >1        A      subscale   1    yes
> >1        A      subscale   2    yes
> >1        B      subscale   3    yes
> >1        B      subscale   4    yes
> >2        A&C    composite  5    yes
> >3        G&F    composite  6    yes
> >4        E      subscale   7    yes
> >4        F      subscale   8    yes
> >4        E&F    composite  9    no
> >
> >On Wed, Nov 24, 2021 at 12:59 PM Viechtbauer, Wolfgang (SP)
> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >>
> >> Let me use a concrete example.
> >>
> >> Say I have studies assessing the effectiveness of a treatment on depression.
> >Some studies report means and SDs of the treated and control groups for
> >overall/composite scales such as the BDI, HAM-D, CES-D, and so on. For such a
> >study, I would compute its effect size based on whatever scale it used.
> >>
> >> Studies may also have used multiple such scales. Then I would also compute
> >multiple effect sizes, one per scale. Of course, I would then have to take the
> >dependency of multiple effect sizes computed based on the same sample into
> >consideration.
> >>
> >> Say there are also some studies that, for some reason, have broken down such a
> >scale into a few subscales, say BDI1 and BDI2, and they do not report means and
> >SDs for the overall BDI scale, only for these subscales.
> >>
> >> I would then compute effect sizes based on BDI1 and BDI2 and again, accounting
> >for their dependency, include them in the same analysis as all of the above.
> >>
> >> I personally see no major issues with this. BDI is a mixture of BDI1 and BDI2
> >anyway, so if I only have BDI, then this is what the effect size reflects. If I
> >include effect sizes based on BDI1 and BDI2 in the analysis, then the model
> >essentially mixes them together.
> >>
> >> Scales may also measure multiple inherently different types of outcomes, such
> >as the HADS, which has subscales for anxiety and depression. Not sure if it
> >common practice to ever report an overall mean for both of these outcome types
> >together. If both outcome types are of interest (and not just depression), then I
> >can again include both effect sizes (for depression and anxiety) in the same
> >analysis (again, with their covariance, blah blah blah). Plus I'll need a
> >moderator to distinguish the two outcome types. Not sure what I would do with a
> >study that only reports an overall HADS score for the two groups (if this is ever
> >done). I might still include this in the analysis and code the outcome type
> >moderator with a third category for 'mixture'.
> >>
> >> If there are moderators that I want to examine, then I would be inclined to
> >allow for separate relationships for different outcome types. I probably would
> >not examine if the relationship differs for effect sizes that are based on
> >subscales for the same outcome type versus effect sizes that are based on overall
> >measures. Same goes with the random effects structure. But that would be my
> >approach and one could of course separate things further.
> >>
> >> Best,
> >> Wolfgang
> >>
> >> >-----Original Message-----
> >> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> >> >Sent: Wednesday, 24 November, 2021 19:36
> >> >To: Viechtbauer, Wolfgang (SP)
> >> >Cc: R meta
> >> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> >> >subscales
> >> >
> >> >So, you think there is no need to keep everything (i.e., fixed and
> >> >random) separate between studies that only contribute composite and
> >> >studies that only contribute separate subscales?
> >> >
> >> >If there is no need, and both types of studies can be in one model,
> >> >then methodologically, wouldn't it be mixing apples (different
> >> >subscales) and oranges (different composites) in one model?
> >> >
> >> >Thanks,
> >> >Tim M
> >> >
> >> >On Wed, Nov 24, 2021 at 12:22 PM Viechtbauer, Wolfgang (SP)
> >> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >> >>
> >> >> >-----Original Message-----
> >> >> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> >> >> >Sent: Wednesday, 24 November, 2021 19:18
> >> >> >To: Viechtbauer, Wolfgang (SP)
> >> >> >Cc: R meta
> >> >> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> >> >> >subscales
> >> >> >
> >> >> >>rma.mv(es ~ reporting:X1, vi, random = list(~1| study, ~ reporting |
> >> >> >>obs), struct = "DIAG", subset = include == "yes")
> >> >> >
> >> >> >Not sure what X1 is, but yes, this could be a plausible model,
> >> >> >allowing for different within-study variances for 'subscale' versus
> >> >> >'composite' estimates.
> >> >> >
> >> >> >>>>>X1 is a moderator but I think I should keep X1 separate between studies
> >> >for
> >> >> >which we have used their composite result and studies for which we have
> >used
> >> >> >their subscale results, no?
> >> >>
> >> >> That's up to you or one could empirically examine if the association between
> >X1
> >> >and es is different for the two types.
> >> >>
> >> >> Best,
> >> >> Wolfgang