[R-meta] Multilevel meta-analysis of pre-post design studies without control groups

[Ivan Jukic]

Dear Wolfgang and R Meta-analysis SIG community,

First of all, thank you all who contribute with their knowledge (and questions) to this SIG! It's so easy to "lose" hours just scrolling through the archives and learning from all of you. This is my first post, so please let me know if I did something wrong with regards to posting (I'm not handling plain text emails well).

There are a few questions/considerations that I would like to get your thoughts on. I'll briefly explain my case, and then I'll outline my questions at the end.

I'm about to meta-analyse the effects of different fatigue thresholds on some neuromuscular adaptations. All studies, included for the quantitative part, evaluated and compared the effects of different fatigue thresholds (PRE vs POST). These thresholds within the studies were differing only in their magnitude. However, not a single study included a control group because the aim of every study was to quantify the amount of fatigue necessary to elicit some physiological adaptations. Moreover, groups in these studies were independent (i.e., participants were members of only one fatigue threshold group in a given study). Some studies compared 4, some 3, some 2, and two studies only evaluated the effects of one fatigue threshold. In this regard, I thought about using hierarchical multilevel meta-regression to examine whether these thresholds affect the rate of neuromuscular adaptations. I managed to obtain pre to post correlations from ~85% of the studies. I've been looking around on stack and cross validated, but haven't seen anyone dealing with this kind of design because it's very rare and probably not meaningful in many situations (while in this one, it is). I planned to approach the calculation of the effect size using the methods outlined by Becker (1988) and Morris (2002, 2008). Wolfgang wrote an excellent post on his website illustrating the procedures as well. I used these procedures in previous metas when I had control groups but calculations should remain the same even now in my opinion. Small subset of the data necessary for you to get the idea about it is below (didn't know a better way of doing this with plain text).

----

structure(list(study_id = c(1L, 1L, 2L, 2L, 2L, 2L, 3L, 3L, 4L, 
4L, 4L, 4L, 5L, 6L, 6L, 7L, 7L), es_id = c(1L, 2L, 3L, 4L, 5L, 
6L, 8L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L), VL = c(15, 
30, 20, 20, 20, 20, 5, 20, 10, 10, 10, 10, 20, 20, 40, 15, 30
), yi = structure(c(1.26329764453676, 1.92190659061828, 0.490288005035358, 
0.473318654746664, 0.438876730850066, 0.316811228214484, 0.724490595454125, 
0.888808062040681, 3.27252802856037, 1.0284768756425, 0.645228131946896, 
0.635333041610322, 0.27735643455418, 1.42543020053664, 0.8533533757798, 
0.425203519630572, 0.275649178243267), vi = c(0.109796046934607, 
0.214686247153099, 0.0181488954925969, 0.0186269093081995, 0.0162132990551026, 
0.00982308464517312, 0.0334962207633824, 0.0519222988903274, 
0.514065440805146, 0.106262031078698, 0.0480145155570764, 0.0256203669891738, 
0.0242052746578586, 0.114493802358414, 0.0676105991977391, 0.0387998770691392, 
0.0237489043416368)), row.names = c(NA, -17L), class = "data.frame")

----

1)  Considering the nested structure of the data, my approach to this would be 
rma.mv(yi = yi, V = vi, random = ~ 1 | study_id/es_id, mods = ~ VL). Note, "VL" represents fatigue threshold.

I'm wondering what is your opinion on my approach to ES calculation and data synthesis in general?

2) Different tasks were performed in different studies. However, the task was always consistent within a given study (i.e., it was the same for all fatigue thresholds groups in a given study), except for two studies which used 4 tasks (study_id 2 and 4 in the data above). Now, since these two studies were the only ones to complicate everything while having 4 tasks within one fatigue threshold group, I was thinking about averaging these effect sizes instead of complicating the model further while accounting for it. The task will be considered as a moderator anyway. What is your opinion on this?

3) My main moderator in this case is fatigue threshold (i.e., VL). However, there are also some other moderators that I would like to control for. After including them, let's say that we realise that some of them are significant, while the main moderator isn't. Since this tells us that the magnitude of the fatigue threshold does not affect the rates of neuromuscular adaptations, I'm wondering how meaningful it is for us to discuss other, significant moderators which were included in order to control for them rather to examine their effects on chronic adaptations. Note that these are some physiological covariates for which it is expected to affect the magnitude of adaptation. I would appreciate your thoughts on this.

----

4) This one is less related to my specific design, but more to multilevel meta-analysis in general. I'm aware that there is no consensus on this matter, but how many studies is generally recommended (i.e., a rule of thumb according to you) to have per moderator? I've seen recommendations of 10, 15, and even 5-6. However, in the context of multilevel meta-analysis, would this rather correspond to the number of effect sizes? 

I really appreciate you taking the time to read all of this, and I'm looking forward to your inputs.

Cheers,
Ivan