[R-meta] Multilevel Meta-regression with Multiple Level Covariates

[James Pustejovsky]

Hi Billy,

The approach you describe seems reasonable to me. Whether you've got too
many predictors will depend on how many studies' worth of data you are able
to gather and on the distribution of the item-level, study-level, and
test-level characteristics. Not knowing the correlations between items does
complicate things a bit, but could be handled using robust variance
estimation methods.

One way to set up this problem might be by using proportion of successes as
the effect size metric, modeled by a binomial distribution (where the
number of trials = number of respondents to that item), and where the
probability of success is related to the covariates via a logit (or probit)
link. Combining this with RVE for standard errors gives you something like
a GEE model.

James

On Fri, Sep 25, 2020 at 12:39 PM Billy Goette <billy.goette using gmail.com>
wrote:

> Hope this finds everyone well,
>
> I was hoping to get some feedback on addressing some practical concerns
> with an idea for a meta-analysis that I have. At the core, I'm concerned
> with the interpretability of the data analysis, so I'm trying to check
> whether this is even possible before I start the study.
>
> My field commonly administers lists of words that a participant must read
> correctly. Item difficulty is often summarized in studies as the proportion
> of respondents who got an item correct over the total sample size. Studies
> use different word-reading tests and lists of varying length (usually
> between 20 and 50). Each test differs in how long the words are
> presented, how many words are presented at a time, etc. I'm interested in
> whether certain word-specific variables (e.g., frequency of word in
> English) are related to the item difficulty (i.e., proportion of people who
> answer it correctly).
>
> My current conceptualization of the problem is as a multilevel
> meta-regression. The effect size is computed from the proportion of correct
> answers to each word which is nested within study and within tests. I would
> have multiple covariates (i.e., word traits) for each effect size,
> covariates at the study level (i.e., average sample demographics), and
> covariates at the test level (i.e., administration differences). My concern
> is that these are too many predictors to include in a meta-regression, and
> I also am anticipating that I won't have any information about
> the covariance matrix of the effect sizes. Any recommendations, words of
> warning, caveats, or suggestions would be incredibly helpful since I'm
> early in the process.
>
> Thank you!
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]