[R-meta] multiple models in one study

[Michael Dewey]

Dear Valeria

Comment in-line

On 27/03/2023 10:05, Valeria Ivaniushina via R-sig-meta-analysis wrote:
> Hi James
> 
> You have perfectly described my problem of model selection: while the
> dependent variable and key predictors are the same, sets of control
> variables are different -- within each paper as well as between papers.
> 
> You ask two questions at the end (choosing between the results that are
> closest to
> the ideal and the results that are most comparable across studies; include
> or exclude model without control variables).  I guess I could try this and
> that and then compare results, what do you think?
> 

Before you do that I suggest asking yourself what you will do if the 
results are the same, or more challengingly, if they are different. If 
you could write a scientifically convincing explanation of the 
diferences then fine but otherwise you may just ed up scratching your head.

Michael


> Best,
> Valeria
> 
> On Fri, Mar 24, 2023 at 6:58 PM James Pustejovsky via R-sig-meta-analysis <
> r-sig-meta-analysis using r-project.org> wrote:
> 
>> Hi Valeria,
>>
>> It sounds like you're interested in synthesizing sets of regression
>> coefficients and the issue is that some papers report multiple regression
>> specifications that fit your criteria. For instance, a paper might report
>> three models:
>> Model 1: Y = b0 + b1 A + b2 B + b3 C
>> Model 2: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F
>> Model 3: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F + <a bunch of
>> other stuff>
>> And perhaps you're just interested in analyzing the coefficients (b1, b2,
>> b3). Does this description track with what you're wondering about?
>>
>> If so, then the challenge is that the definition of regression coefficients
>> depends on ALL of the variables in the model, so the coefficients (b1, b2,
>> b3) from Model 1 aren't really estimating the same parameters as the (b1,
>> b2, b3) from Model 3. From your research aims and inclusion criteria, is it
>> possible to define an "ideal analysis" that most closely matches the
>> questions you're trying to investigate? If so, then perhaps you can select
>> results from each study that come closest to matching the ideal analysis.
>> This would be pretty similar to the "best-set" strategy.
>>
>> Another thing to consider is how similar the regression specifications are
>> across studies. For example, say that you've got 10 studies meeting
>> inclusion criteria. For the first 8 studies, the specification that most
>> closely matches your ideal analysis is Model 2. But then for study 9, the
>> only thing that's reported is Model 1 and for study 10, Models 1, 2, and 3
>> are all reported and Model 3 is the one that most closely matches your
>> ideal analysis. For study 10, should you take the coefficients from Model 2
>> or Model 3? The tension is between choosing the results that are closest to
>> the ideal or choosing the results that are most comparable across all
>> included studies. And for study 1, should you take the results from Model 1
>> or just exclude it entirely because it doesn't report a specification that
>> controls for factors D, E, and F?
>>
>> James
>>
>>
>> On Thu, Mar 23, 2023 at 3:17 AM Viechtbauer, Wolfgang (NP) via
>> R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>
>>> Dear Valeria,
>>>
>>> I would say this depends on the aims. If there is one key predictor of
>>> interest, then I would focus on that. If that's not the case, then I
>> would
>>> extract all the ones that are of interest. Taking an average of all
>>> coefficients (if this is what the "average-set" approach entails) doesn't
>>> make much sense to me unless they are all measuring the same construct in
>>> the same direction and in the same units (all unlikely).
>>>
>>> If you extract multiple coefficients, you of course have to account for
>>> the fact that they are not independent.
>>>
>>> Best,
>>> Wolfgang
>>>
>>>> -----Original Message-----
>>>> From: R-sig-meta-analysis [mailto:
>>> r-sig-meta-analysis-bounces using r-project.org] On
>>>> Behalf Of Valeria Ivaniushina via R-sig-meta-analysis
>>>> Sent: Wednesday, 22 March, 2023 17:52
>>>> To: R meta
>>>> Cc: Valeria Ivaniushina
>>>> Subject: [R-meta] multiple models in one study
>>>>
>>>> Hi
>>>>
>>>> I want to perform a meta-analysis of the relation between the outcome
>> and
>>>> key explanatory variables expressed as regression coefficients.
>>>>
>>>> As a rule, authors report several models with different specifications.
>> I
>>>> wonder which regression coefficients should I collect?
>>>>
>>>> In the book Meta-regression analysis in economics and business (Stanley
>> &
>>>> Doucouliagos, 2012)
>>>> several approaches are described:
>>>> - The best-set = ONE estimate from each study, using the KEY regression
>>> >from each paper
>>>> - The average-set = an average of all coefficients reported in the study
>>>> - The all-set = all relevant estimates reported in the study
>>>>
>>>> Which approach is preferable? Are there additional considerations that I
>>>> have to take into account?
>>>>
>>>> Regards,
>>>> Valeria
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> To manage your subscription to this mailing list, go to:
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html