[R-meta] rma.mv for studies reporting composite of and/or individual subscales

[Viechtbauer, Wolfgang (SP)]

Let me use a concrete example.

Say I have studies assessing the effectiveness of a treatment on depression. Some studies report means and SDs of the treated and control groups for overall/composite scales such as the BDI, HAM-D, CES-D, and so on. For such a study, I would compute its effect size based on whatever scale it used.

Studies may also have used multiple such scales. Then I would also compute multiple effect sizes, one per scale. Of course, I would then have to take the dependency of multiple effect sizes computed based on the same sample into consideration.

Say there are also some studies that, for some reason, have broken down such a scale into a few subscales, say BDI1 and BDI2, and they do not report means and SDs for the overall BDI scale, only for these subscales.

I would then compute effect sizes based on BDI1 and BDI2 and again, accounting for their dependency, include them in the same analysis as all of the above.

I personally see no major issues with this. BDI is a mixture of BDI1 and BDI2 anyway, so if I only have BDI, then this is what the effect size reflects. If I include effect sizes based on BDI1 and BDI2 in the analysis, then the model essentially mixes them together.

Scales may also measure multiple inherently different types of outcomes, such as the HADS, which has subscales for anxiety and depression. Not sure if it common practice to ever report an overall mean for both of these outcome types together. If both outcome types are of interest (and not just depression), then I can again include both effect sizes (for depression and anxiety) in the same analysis (again, with their covariance, blah blah blah). Plus I'll need a moderator to distinguish the two outcome types. Not sure what I would do with a study that only reports an overall HADS score for the two groups (if this is ever done). I might still include this in the analysis and code the outcome type moderator with a third category for 'mixture'.

If there are moderators that I want to examine, then I would be inclined to allow for separate relationships for different outcome types. I probably would not examine if the relationship differs for effect sizes that are based on subscales for the same outcome type versus effect sizes that are based on overall measures. Same goes with the random effects structure. But that would be my approach and one could of course separate things further.

Best,
Wolfgang

>-----Original Message-----
>From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
>Sent: Wednesday, 24 November, 2021 19:36
>To: Viechtbauer, Wolfgang (SP)
>Cc: R meta
>Subject: Re: rma.mv for studies reporting composite of and/or individual
>subscales
>
>So, you think there is no need to keep everything (i.e., fixed and
>random) separate between studies that only contribute composite and
>studies that only contribute separate subscales?
>
>If there is no need, and both types of studies can be in one model,
>then methodologically, wouldn't it be mixing apples (different
>subscales) and oranges (different composites) in one model?
>
>Thanks,
>Tim M
>
>On Wed, Nov 24, 2021 at 12:22 PM Viechtbauer, Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>
>> >-----Original Message-----
>> >From: Timothy MacKenzie [mailto:fswfswt using gmail.com]
>> >Sent: Wednesday, 24 November, 2021 19:18
>> >To: Viechtbauer, Wolfgang (SP)
>> >Cc: R meta
>> >Subject: Re: rma.mv for studies reporting composite of and/or individual
>> >subscales
>> >
>> >>rma.mv(es ~ reporting:X1, vi, random = list(~1| study, ~ reporting |
>> >>obs), struct = "DIAG", subset = include == "yes")
>> >
>> >Not sure what X1 is, but yes, this could be a plausible model,
>> >allowing for different within-study variances for 'subscale' versus
>> >'composite' estimates.
>> >
>> >>>>>X1 is a moderator but I think I should keep X1 separate between studies
>for
>> >which we have used their composite result and studies for which we have used
>> >their subscale results, no?
>>
>> That's up to you or one could empirically examine if the association between X1
>and es is different for the two types.
>>
>> Best,
>> Wolfgang