[R-meta] Studies with more than one control group

[Jack Solomon]

Dear James,

Thank you for your reply. "controlID" distinguishes between effect sizes
(SMDs in this case) that have been obtained by comparing the treated groups
to control 1 vs. control 2 (see below).

I was wondering if adding such an ID variable (just like schoolID) and the
random effect associated with it would also mean that we are generalizing
beyond the levels of controlID, which then, would mean that we anticipate
that each study 'could' have any number of control groups and not just
limited to a max of 2?

Thanks again, Jack

studyID  yi  controlID
1        .1      1
1        .2      2
1        .3      1
1        .4      2
2        .5      1
2        .6      2
3        .7      1

On Wed, Jul 21, 2021 at 10:13 AM James Pustejovsky <jepusto using gmail.com>
wrote:

> Hi Jack,
>
> To make sure I follow the structure of your data, let me ask: Do controlID
> = 1 or controlID = 2 correspond to specific *types* of control groups that
> have the same meaning across all of your studies? Or is this just an
> arbitrary ID variable?
>
> In my earlier response, I was assuming that controlID in your data is just
> an ID variable. Using random effects specified as
> ~ | studyID/controlID
> means that you're including random *intercept* terms for each unique
> control group nested within studyID. It has nothing to do with the number
> of control groups.
>
> James
>
>
>
> On Mon, Jul 19, 2021 at 10:56 PM Jack Solomon <kj.jsolomon using gmail.com>
> wrote:
>
>> Dear James,
>>
>> I'm coming back to this after a while (preparing the data). A quick
>> follow-up. So, you mentioned that if I have several studies that have used
>> more than 1 control group (in my data up to 2), I can possibly add a
>> random-effect (controlID) to capture any heterogeneity in the effect sizes
>> across control groups nested within studies.
>>
>> My question is that adding a controlID random-effect (a binary indicator:
>> 1 or 2) would also mean that we intend to generalize beyond the possible
>> number of control groups that a study can employ (for my data beyond 2
>> control groups)?
>>
>> Thank you,
>> Jack
>>
>> On Thu, Jun 24, 2021 at 4:52 PM Jack Solomon <kj.jsolomon using gmail.com>
>> wrote:
>>
>>> Thank you very much for the clarification. That makes perfect sense.
>>>
>>> Jack
>>>
>>> On Thu, Jun 24, 2021 at 4:44 PM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>>
>>>> The random effect for controlID is capturing any heterogeneity in the
>>>> effect sizes across control groups nested within studies, *above and beyond
>>>> heterogeneity explained by covariates.* Thus, if you include a covariate to
>>>> distinguish among types of control groups, and the differences between
>>>> types of control groups are consistent across studies, then the covariate
>>>> might explain all (or nearly all) of the variation at that level, which
>>>> would obviate the purpose of including the random effect at that level.
>>>>
>>>> On Thu, Jun 24, 2021 at 9:56 AM Jack Solomon <kj.jsolomon using gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you James. On my question 3, I was implicitly referring to my
>>>>> previous question (a previous post titled: Studies with independent
>>>>> samples) regarding the fact that if I decide to drop 'sampleID', then I
>>>>> need to change the coding of the 'studyID' column (i.e., then, each sample
>>>>> should be coded as an independent study). So, in my question 3, I really
>>>>> was asking that in the case of 'controlID', removing it doesn't require
>>>>> changing the coding of any other columns in my data.
>>>>>
>>>>> Regarding adding 'controlID' as a random effect, you said: "... an
>>>>> additional random effect for controlID will depend on how many studies
>>>>> include multiple control groups and whether the model includes a covariate
>>>>> to distinguish among types of control groups (e.g., business-as-usual
>>>>> versus waitlist versus active control group)."
>>>>>
>>>>> I understand that the number of studies with multiple control groups
>>>>> is important in whether to add a random effect or not. But why having "a
>>>>> covariate to distinguish among types of control groups" is important in
>>>>> whether to add a random effect or not?
>>>>>
>>>>> Thanks, Jack
>>>>>
>>>>> On Thu, Jun 24, 2021 at 9:17 AM James Pustejovsky <jepusto using gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jack,
>>>>>>
>>>>>> Responses inline below.
>>>>>>
>>>>>> James
>>>>>>
>>>>>>
>>>>>>> I have come across a couple of primary studies in my meta-analytic
>>>>>>> pool
>>>>>>> that have used two comparison/control groups (as the definition of
>>>>>>> 'control' has been debated in the literature I'm meta-analyzing).
>>>>>>>
>>>>>>> (1) Given that, should I create an additional column ('control') to
>>>>>>> distinguish between effect sizes (SMDs in this case) that have been
>>>>>>> obtained by comparing the treated groups to control 1 vs. control 2
>>>>>>> (see
>>>>>>> below)?
>>>>>>>
>>>>>>>
>>>>>> Yes. Along the same lines as my response to your earlier question, it
>>>>>> seems prudent to include ID variables like this in order to describe the
>>>>>> structure of the included studies.
>>>>>>
>>>>>>
>>>>>>> (2) If yes, then, does the addition of a 'control' column call for
>>>>>>> the
>>>>>>> addition of a random effect for 'control' of the form:  "~ |
>>>>>>> studyID/controlID" (to be empirically tested)?
>>>>>>>
>>>>>>>
>>>>>> I expect you will find differences of opinion here. Pragmatically,
>>>>>> the feasibility of estimating a model with an additional random effect for
>>>>>> controlID will depend on how many studies include multiple control groups
>>>>>> and whether the model includes a covariate to distinguish among types of
>>>>>> control groups (e.g., business-as-usual versus waitlist versus active
>>>>>> control group).
>>>>>>
>>>>>> At a conceptual level, omitting random effects for controlID leads to
>>>>>> essentially the same results as averaging the ES across both control
>>>>>> groups. If averaging like this makes conceptual sense, then omitting the
>>>>>> random effects might be reasonable.
>>>>>>
>>>>>>
>>>>>>> (3) If I later decide to drop controlID from my dataset, I think I
>>>>>>> can
>>>>>>> still keep all effect sizes from both control groups intact without
>>>>>>> any
>>>>>>> changes to my coding scheme, right?
>>>>>>>
>>>>>>
>>>>>> I don't understand what you're concern is here. Why not just keep
>>>>>> controlID in your dataset as a descriptor, even if it doesn't get used in
>>>>>> the model?
>>>>>>
>>>>>

	[[alternative HTML version deleted]]