[R-meta] Multilevel Meta-regression with Multiple Level Covariates

[James Pustejovsky]

Hi Billy,

The metafor package does have the rma.glmm() function for fitting
generalized linear mixed models (GLMMs), including with non-gaussian
distributions. However, as far as I understand GLMMs (which are also what
lme4 does) would not really work for your data because you have cluster
dependence in the measurement level of the model, not just in the random
effects. GEEs could handle this though, and the packages you found (gee and
geepack) are what I would recommend using. There's a very large literature
on these models. Here's a good introductory blog post, which has links to
further (more formal) reading:
https://rlbarter.github.io/Practical-Statistics/2017/05/10/generalized-estimating-equations-gee/

James

On Tue, Sep 29, 2020 at 4:27 PM Billy Goette <billy.goette using gmail.com> wrote:

> This is an interesting framework (GEE that is) that I was not previously
> familiar with. I appreciate bringing my attention to this. I agree that a
> binomial distribution is more intuitive to what I'm trying to answer. If I
> had actual, item-level data for a sample (i.e., each participant's
> responses -- either 0 or 1 -- to each item), then I would use GLMM assuming
> a binomial, though really Bernoulli, distribution to do an item response
> theory model. The idea of doing this meta-analysis actually came from
> trying to figure out how to do that kind of analysis on summary level data
> since I don't have that individual-level data.
>
> For my own edification, is this something that is doable in metafor? I
> don't see how to specify a non-gaussian distribution in rma.mv, but I'm
> also no expert on the metafor package. Cursory googling brought my
> attention to gee and geepack packages for R, but it doesn't seem like it's
> possible to include more than one cluster in those packages. I could do
> this in lme4, but I believe that their estimation method is still
> inappropriate for meta-analysis (though maybe this has been changed? it's
> been a while since I looked at multilevel meta-analysis in lme4 since
> rma.mv works so well).
>
> Also, is there any guidance on how many studies should be collected for
> these kinds of analyses? I suspect that this is a relatively specific
> meta-analysis, and it sounds like rules-of-thumbs will be biased if the
> distribution of moderators is restricted or sparse. Any good Monte Carlo
> simulation methods that could be used to check the extent to which the
> model has too few studies to estimate the various moderators?
>
> On Mon, Sep 28, 2020 at 8:17 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Billy,
>>
>> The approach you describe seems reasonable to me. Whether you've got too
>> many predictors will depend on how many studies' worth of data you are able
>> to gather and on the distribution of the item-level, study-level, and
>> test-level characteristics. Not knowing the correlations between items does
>> complicate things a bit, but could be handled using robust variance
>> estimation methods.
>>
>> One way to set up this problem might be by using proportion of successes
>> as the effect size metric, modeled by a binomial distribution (where the
>> number of trials = number of respondents to that item), and where the
>> probability of success is related to the covariates via a logit (or probit)
>> link. Combining this with RVE for standard errors gives you something like
>> a GEE model.
>>
>> James
>>
>> On Fri, Sep 25, 2020 at 12:39 PM Billy Goette <billy.goette using gmail.com>
>> wrote:
>>
>>> Hope this finds everyone well,
>>>
>>> I was hoping to get some feedback on addressing some practical concerns
>>> with an idea for a meta-analysis that I have. At the core, I'm concerned
>>> with the interpretability of the data analysis, so I'm trying to check
>>> whether this is even possible before I start the study.
>>>
>>> My field commonly administers lists of words that a participant must read
>>> correctly. Item difficulty is often summarized in studies as the
>>> proportion
>>> of respondents who got an item correct over the total sample size.
>>> Studies
>>> use different word-reading tests and lists of varying length (usually
>>> between 20 and 50). Each test differs in how long the words are
>>> presented, how many words are presented at a time, etc. I'm interested in
>>> whether certain word-specific variables (e.g., frequency of word in
>>> English) are related to the item difficulty (i.e., proportion of people
>>> who
>>> answer it correctly).
>>>
>>> My current conceptualization of the problem is as a multilevel
>>> meta-regression. The effect size is computed from the proportion of
>>> correct
>>> answers to each word which is nested within study and within tests. I
>>> would
>>> have multiple covariates (i.e., word traits) for each effect size,
>>> covariates at the study level (i.e., average sample demographics), and
>>> covariates at the test level (i.e., administration differences). My
>>> concern
>>> is that these are too many predictors to include in a meta-regression,
>>> and
>>> I also am anticipating that I won't have any information about
>>> the covariance matrix of the effect sizes. Any recommendations, words of
>>> warning, caveats, or suggestions would be incredibly helpful since I'm
>>> early in the process.
>>>
>>> Thank you!
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>

	[[alternative HTML version deleted]]