[R-meta] Questions about model averaging with complex multilevel meta-analytic model

[Viechtbauer, Wolfgang (NP)]

Dear Maggie,

Some notes on this:

1) dredge() (and I assume the other functions from MuMIn as well) examines the 'nobs' attribute that logLik() returns to determine the sample size of the various models. However, when using REML estimation, nobs = k - p, where p is the number of model coefficients (for some technical reasons that are not important right now). However, this leads dredge() to think that the sample size differs across models where p differs.

In general, you should use method="ML" when comparing models that differ in terms of their fixed effects.[1] In that case, nobs = k and this issue won't arise.

2) I would recommend to do all transformations (like mean centering or things like sqrt(vi)) outside of the model call (so, beforehand).

3) You have *a lot* of fixed effects and even interactions. This will lead to many models that dredge() needs to fit. This could take a looooong time. dredge() has a 'cluster' argument for doing parallel processing, which you might consider using if you have powerful enough hardware. Still, even then this could be a rather daunting task.

4) I can confirm that dredge() works just fine with rma.mv() models. An example with a similar model as you are fitting can be found here:

https://gist.github.com/wviechtb/891483eea79da21d057e60fd1e28856b

Best,
Wolfgang

[1] Actually, based on some research we did, REML might actually work: 

Cinar, O., Umbanhowar, J., Hoeksema, J. D., & Viechtbauer, W. (2021). Using information-theoretic approaches for model selection in meta-analysis. Research Synthesis Methods, 12(4), 537–556. https://doi.org/10.1002/jrsm.1489

But we didn't examine complex models like you are using and I would still be very cautious with using REML when doing so.

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Margaret Slein
>Sent: Saturday, 24 September, 2022 18:58
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Questions about model averaging with complex multilevel meta-
>analytic model
>
>Hi Wolfgang,
>
>I am PhD student at UBC in Vancouver, Canada, currently working on a meta-
>analysis. I have been trying to do model selection and model averaging using the
>model.sel() and model.avg() functions from the MuMIn package with an rma.mv model
>object, while also following your metafor help page using MuMIn and glmulti. I
>have been unable to get any of my models to perform model selection or model
>averaging because each model is being fit to different data. I have ensured there
>are no missing values or NAs across the data frame.
>
>Is it possible to do model averaging with the rma.mv function with 3 level random
>effects, a phylogenetic correlation, and several moderators? There are currently
>no examples I could find using model selection or averaging with this model
>structure and I have had no luck on stack overflow.
>
>Here is the full model I am trying to run:
>
>full_mod<-rma.mv(yi=yi, V=V,
>                  mods = ~ I(flux_range-mean(flux_range))*I(mean_temp_constant-
>mean(mean_temp_constant))
>                  +I(flux_range-mean(flux_range))*experiment_type
>                  +I(mean_temp_constant-mean(mean_temp_constant))*experiment_type
>                  +I(secondary_temp - mean(secondary_temp))*experiment_type
>                  +duration_standard*experiment_type
>                  +experiment_type*experiment_type
>                  +exp_age*experiment_type
>                  +size*experiment_type
>                  +ecosystem*experiment_type
>                  +trait_directionality*experiment_type
>                  +sqrt(vi)*experiment_type, dfs="contain",
>                  random = list( ~1 | phylo, ~1 | study_id, ~1 | response_id),
>                  R = list(phylo=cor), test="t",
>                  method="REML", data=dat_ES_final_2)
>
>I have also tried using dredge() in MuMIn in addition to trying my own subset of
>models with no luck but it takes days for dredge() to iterate and still yields an
>error when trying to perform model averaging that none of the models are being
>fit to the same data. Any suggestions or assistance you can provide on this would
>be greatly appreciated.
>
>Cheers,
>Maggie
>
><*)))><>  <*)))><>   <*)))><>  <*)))><>
>
>Maggie Slein (she/her/hers)
>PhD Student, O’Connor Lab
>Department of Zoology
>Unceded xʷməθkʷəy̓əm (Musqueam) territory
>University of British Columbia