[R-meta] rma.mv for studies reporting composite of and/or individual subscales

[Timothy MacKenzie]

Note:
by unique composites (e.g., AB;  AC;  ABC): I mean a study made its
composite out of A&B; another study made its composite out of A&C, and
another study made its composite out of A&B&C etc

by reporting subscales separately (e.g., A,B;  A,C;  A,B,C): I mean a
study separately reported A and B; another study separately reported A
and C, and another study separately reported A and B and C

On Wed, Nov 24, 2021 at 3:10 PM Timothy MacKenzie <fswfswt using gmail.com> wrote:
>
> By standard view, I mean the usual/standard way in which construct X
> should be measured (i.e., using a scale whose subscales are A, B, C).
>
> Under this standard view, when we look at the literature, we see:
>
> 1- Some studies mixed and matched the usual subscales to create their
> unique composites (e.g., AB;  AC;  ABC)
> 2- Some studies report some or all these usual subscales and report
> each separately (e.g., A,B;  A,C;  A,B,C)
>
> By alternative view, I mean the researcher-constructed ways in which
> construct X can be measured (i.e., using ANY scale whose subscales can
> be ANYTHING appropriate to the researchers e.g., E,F,G ...).
>
> Under this alternative view, when we look at the literature, we see:
>
> (3) Some studies mixed and matched their own subscales to create their
> unique composites (e.g., EF;  EG;  EFG),
> (4) Some studies report some or all such subscales separately (e.g.,
> E,F;  E,G;  E,F,G)
>
> The result of such a trend is the data structure below. Therefore,
> there are three gray areas for me:
>
> 1) Dealing with composite vs separate subscales (resolved:-)
> 2) Dealing with whether effects have been obtained under standard or
> alternative view (maybe this should be a moderator?)
> 3) How should this data be subgrouped i.e., only by composite vs.
> subscales or by standard vs. alternative view?
>
> Thanks, Tim M
>
> study subscale  reporting  obs  include
> 1        A      subscale   1    yes
> 1        A      subscale   2    yes
> 1        B      subscale   3    yes
> 1        B      subscale   4    yes
> 2        A&C    composite  5    yes
> 3        G&F    composite  6    yes
> 4        E      subscale   7    yes
> 4        F      subscale   8    yes
> 4        E&F    composite  9    no
>
> On Wed, Nov 24, 2021 at 2:29 PM Viechtbauer, Wolfgang (SP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >
> > Sorry, I can't follow. What is 'standard view' and 'alternative view'? Those sound the same to me, except for different letters.
> >
> > Best,
> > Wolfgang
> >
> > >-----Original Message-----
> > >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> > >Sent: Wednesday, 24 November, 2021 20:56
> > >To: Viechtbauer, Wolfgang (SP)
> > >Cc: R meta
> > >Subject: Re: rma.mv for studies reporting composite of and/or individual
> > >subscales
> > >
> > >Appreciate it. Thank you very much. My response is below inline.
> > >
> > >Say there are also some studies that, for some reason, have broken
> > >down such a scale into a few subscales, say BDI1 and BDI2, and they do
> > >not report means and SDs for the overall BDI scale, only for these
> > >subscales.
> > >
> > >BDI is a mixture of BDI1 and BDI2 anyway, so if I only have BDI, then
> > >this is what the effect size reflects. If I include effect sizes based
> > >on BDI1 and BDI2 in the analysis, then the model essentially mixes
> > >them together.
> > >
> > >>>>>Sure, but, what if
> > >**one the one hand**: BDI *in standard view* is a mixture of A, B and
> > >C subscales and (1) some studies can mix and match them to create
> > >their unique composites (AB;  AC;  ABC), (2) some studies report some
> > >or all these subscales (A,B;  A,C;  A,B,C), and
> > >
> > >**on the other hand**: BDI *in alternative view* is a mixture of E, F
> > >and G subscales and (3) some studies can mix and match them to create
> > >their unique composites (EF;  EG;  EFG), and (4) some studies report
> > >some or all these subscales (E,F;  E,G;  E,F,G)?
> > >
> > >This is what is reflected in my data structure below (as mentioned
> > >earlier, the number of unique subscales is about the number of
> > >studies).
> > >
> > >Thanks, Tim M
> > >
> > >study subscale  reporting  obs  include
> > >1        A      subscale   1    yes
> > >1        A      subscale   2    yes
> > >1        B      subscale   3    yes
> > >1        B      subscale   4    yes
> > >2        A&C    composite  5    yes
> > >3        G&F    composite  6    yes
> > >4        E      subscale   7    yes
> > >4        F      subscale   8    yes
> > >4        E&F    composite  9    no
> > >
> > >On Wed, Nov 24, 2021 at 12:59 PM Viechtbauer, Wolfgang (SP)
> > ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >>
> > >> Let me use a concrete example.
> > >>
> > >> Say I have studies assessing the effectiveness of a treatment on depression.
> > >Some studies report means and SDs of the treated and control groups for
> > >overall/composite scales such as the BDI, HAM-D, CES-D, and so on. For such a
> > >study, I would compute its effect size based on whatever scale it used.
> > >>
> > >> Studies may also have used multiple such scales. Then I would also compute
> > >multiple effect sizes, one per scale. Of course, I would then have to take the
> > >dependency of multiple effect sizes computed based on the same sample into
> > >consideration.
> > >>
> > >> Say there are also some studies that, for some reason, have broken down such a
> > >scale into a few subscales, say BDI1 and BDI2, and they do not report means and
> > >SDs for the overall BDI scale, only for these subscales.
> > >>
> > >> I would then compute effect sizes based on BDI1 and BDI2 and again, accounting
> > >for their dependency, include them in the same analysis as all of the above.
> > >>
> > >> I personally see no major issues with this. BDI is a mixture of BDI1 and BDI2
> > >anyway, so if I only have BDI, then this is what the effect size reflects. If I
> > >include effect sizes based on BDI1 and BDI2 in the analysis, then the model
> > >essentially mixes them together.
> > >>
> > >> Scales may also measure multiple inherently different types of outcomes, such
> > >as the HADS, which has subscales for anxiety and depression. Not sure if it
> > >common practice to ever report an overall mean for both of these outcome types
> > >together. If both outcome types are of interest (and not just depression), then I
> > >can again include both effect sizes (for depression and anxiety) in the same
> > >analysis (again, with their covariance, blah blah blah). Plus I'll need a
> > >moderator to distinguish the two outcome types. Not sure what I would do with a
> > >study that only reports an overall HADS score for the two groups (if this is ever
> > >done). I might still include this in the analysis and code the outcome type
> > >moderator with a third category for 'mixture'.
> > >>
> > >> If there are moderators that I want to examine, then I would be inclined to
> > >allow for separate relationships for different outcome types. I probably would
> > >not examine if the relationship differs for effect sizes that are based on
> > >subscales for the same outcome type versus effect sizes that are based on overall
> > >measures. Same goes with the random effects structure. But that would be my
> > >approach and one could of course separate things further.
> > >>
> > >> Best,
> > >> Wolfgang
> > >>
> > >> >-----Original Message-----
> > >> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> > >> >Sent: Wednesday, 24 November, 2021 19:36
> > >> >To: Viechtbauer, Wolfgang (SP)
> > >> >Cc: R meta
> > >> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> > >> >subscales
> > >> >
> > >> >So, you think there is no need to keep everything (i.e., fixed and
> > >> >random) separate between studies that only contribute composite and
> > >> >studies that only contribute separate subscales?
> > >> >
> > >> >If there is no need, and both types of studies can be in one model,
> > >> >then methodologically, wouldn't it be mixing apples (different
> > >> >subscales) and oranges (different composites) in one model?
> > >> >
> > >> >Thanks,
> > >> >Tim M
> > >> >
> > >> >On Wed, Nov 24, 2021 at 12:22 PM Viechtbauer, Wolfgang (SP)
> > >> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >> >>
> > >> >> >-----Original Message-----
> > >> >> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
> > >> >> >Sent: Wednesday, 24 November, 2021 19:18
> > >> >> >To: Viechtbauer, Wolfgang (SP)
> > >> >> >Cc: R meta
> > >> >> >Subject: Re: rma.mv for studies reporting composite of and/or individual
> > >> >> >subscales
> > >> >> >
> > >> >> >>rma.mv(es ~ reporting:X1, vi, random = list(~1| study, ~ reporting |
> > >> >> >>obs), struct = "DIAG", subset = include == "yes")
> > >> >> >
> > >> >> >Not sure what X1 is, but yes, this could be a plausible model,
> > >> >> >allowing for different within-study variances for 'subscale' versus
> > >> >> >'composite' estimates.
> > >> >> >
> > >> >> >>>>>X1 is a moderator but I think I should keep X1 separate between studies
> > >> >for
> > >> >> >which we have used their composite result and studies for which we have
> > >used
> > >> >> >their subscale results, no?
> > >> >>
> > >> >> That's up to you or one could empirically examine if the association between
> > >X1
> > >> >and es is different for the two types.
> > >> >>
> > >> >> Best,
> > >> >> Wolfgang