[R-meta] How to interpret when the results from model-based standard errors and robust variance estimation do not corroborate with each other

[Akifumi Yanagisawa]

Hello everyone, 

I would like to ask about meta-analysis with robust variance estimation. I am having difficulty interpreting the predictor variables that are significant by model-based standard errors but are not significant after applying robust variance estimation (RVE).

I am fitting my dataset with three level meta-regression with the metafor package (with study being the clustering variable). In order to deal with the dependency of effect sizes within each study (i.e., the same participants tested repeatedly), I am applying RVE with the clubSandwich package (using coef_test function with the estimator being CR2). [Thank you for the previous suggestions and guidance on robust variance estimation, Dr. Viechtbauer and Dr. Pustejovsky.]

When conducting moderator analysis, I realized that some of the moderator variables that are determined as ‘significant’ by model-based standard errors turn out to be ‘not significant’ after applying robust variance estimation. 

I do understand the conservative nature of the robust variance estimation; however, some of the non-significant variances are factors that have been strongly supported by a large body of previous literature and are actually observed by most of the individual studies. So, in order to carefully interpret the results, I would like to know situations when we have to be careful about a potential Type II error using cluster robust variance estimation. (e.g., potentially difficult to test within study variables even when combining with multilevel meta-analysis?)

If a variable is not significant by RVE, does this just indicate that ‘the null hypothesis’ was not rejected? Or, can we further interpret this discrepancy between RVE and model-based approach in a more informational manner? For example, would it be possible to provide concrete suggestions for other researchers about things that they should focus on or care about as to further testing the potential effect of moderator variables that are not significant by the current meta-analysis?

I would really appreciate it if someone would explain (1) when cluster robust variance estimation potentially increases Type II error rates, and (2) how to interpret when the results from model-based standard errors and robust variance estimation do not corroborate with each other. 

Thank you very much for your time.

Best regards,
Aki