[R-meta] multiple models in one study

[Valeria Ivaniushina]

Hi James

You have perfectly described my problem of model selection: while the
dependent variable and key predictors are the same, sets of control
variables are different -- within each paper as well as between papers.

You ask two questions at the end (choosing between the results that are
closest to
the ideal and the results that are most comparable across studies; include
or exclude model without control variables).  I guess I could try this and
that and then compare results, what do you think?

Best,
Valeria

On Fri, Mar 24, 2023 at 6:58 PM James Pustejovsky via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Hi Valeria,
>
> It sounds like you're interested in synthesizing sets of regression
> coefficients and the issue is that some papers report multiple regression
> specifications that fit your criteria. For instance, a paper might report
> three models:
> Model 1: Y = b0 + b1 A + b2 B + b3 C
> Model 2: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F
> Model 3: Y = b0 + b1 A + b2 B + b3 C + b4 D + b5 E + b6 F + <a bunch of
> other stuff>
> And perhaps you're just interested in analyzing the coefficients (b1, b2,
> b3). Does this description track with what you're wondering about?
>
> If so, then the challenge is that the definition of regression coefficients
> depends on ALL of the variables in the model, so the coefficients (b1, b2,
> b3) from Model 1 aren't really estimating the same parameters as the (b1,
> b2, b3) from Model 3. From your research aims and inclusion criteria, is it
> possible to define an "ideal analysis" that most closely matches the
> questions you're trying to investigate? If so, then perhaps you can select
> results from each study that come closest to matching the ideal analysis.
> This would be pretty similar to the "best-set" strategy.
>
> Another thing to consider is how similar the regression specifications are
> across studies. For example, say that you've got 10 studies meeting
> inclusion criteria. For the first 8 studies, the specification that most
> closely matches your ideal analysis is Model 2. But then for study 9, the
> only thing that's reported is Model 1 and for study 10, Models 1, 2, and 3
> are all reported and Model 3 is the one that most closely matches your
> ideal analysis. For study 10, should you take the coefficients from Model 2
> or Model 3? The tension is between choosing the results that are closest to
> the ideal or choosing the results that are most comparable across all
> included studies. And for study 1, should you take the results from Model 1
> or just exclude it entirely because it doesn't report a specification that
> controls for factors D, E, and F?
>
> James
>
>
> On Thu, Mar 23, 2023 at 3:17 AM Viechtbauer, Wolfgang (NP) via
> R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>
> > Dear Valeria,
> >
> > I would say this depends on the aims. If there is one key predictor of
> > interest, then I would focus on that. If that's not the case, then I
> would
> > extract all the ones that are of interest. Taking an average of all
> > coefficients (if this is what the "average-set" approach entails) doesn't
> > make much sense to me unless they are all measuring the same construct in
> > the same direction and in the same units (all unlikely).
> >
> > If you extract multiple coefficients, you of course have to account for
> > the fact that they are not independent.
> >
> > Best,
> > Wolfgang
> >
> > >-----Original Message-----
> > >From: R-sig-meta-analysis [mailto:
> > r-sig-meta-analysis-bounces using r-project.org] On
> > >Behalf Of Valeria Ivaniushina via R-sig-meta-analysis
> > >Sent: Wednesday, 22 March, 2023 17:52
> > >To: R meta
> > >Cc: Valeria Ivaniushina
> > >Subject: [R-meta] multiple models in one study
> > >
> > >Hi
> > >
> > >I want to perform a meta-analysis of the relation between the outcome
> and
> > >key explanatory variables expressed as regression coefficients.
> > >
> > >As a rule, authors report several models with different specifications.
> I
> > >wonder which regression coefficients should I collect?
> > >
> > >In the book Meta-regression analysis in economics and business (Stanley
> &
> > >Doucouliagos, 2012)
> > >several approaches are described:
> > >- The best-set = ONE estimate from each study, using the KEY regression
> > >from each paper
> > >- The average-set = an average of all coefficients reported in the study
> > >- The all-set = all relevant estimates reported in the study
> > >
> > >Which approach is preferable? Are there additional considerations that I
> > >have to take into account?
> > >
> > >Regards,
> > >Valeria
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> > To manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]