<div dir="ltr"><div dir="ltr"><div>With apologies for posting in HTML.</div><div><br></div><div>Yes, you seem to be in scenario three. There are some systematic features in studies that have led to the repetition of the same outcome levels in each study. But these features are absent in the data. A starting point might be (to take advantage of the possible boost in precision etc. as a result of correlated REs):</div><div><br></div><div>random = list(~ outcome | study, ~ 1 | es_id), struct = "UN"</div><div><br></div>Yes, es_id encapsulates all the uniquenesses (the combination of the features that I referred to in my previous post) that define a single row. If you already have coded for all these features along the way, then following them takes you right to a single row.<div><br></div><div>For example, imagine you have 40 studies, each with two treatment groups (and you have coded for this), and two outcomes (and you have coded for this).</div><div>Thus, you have 40 four-row studies. Here, knowing the study number, group number, and the outcome indicator gets you to a single row in the data.</div><div><br></div><div>That is, the combination of study 1, group 1, outcome A gets you to the first row, 1,1,B gets you to the second one, 1,2,A to the third one, and 1,2,B to the fourth one.</div><div><div style="text-align:center"><img src="cid:ii_kv1w0lh00" alt="image.png" width="211" height="217" style="margin-right:0px"><br></div></div><div style="text-align:left"><br></div><div style="text-align:left">In this case, use of row_id is simply pointless, because you already have all that is needed to travel the distance from a study-level true effect up to a single row's true effect.</div><div style="text-align:left"><br></div><div style="text-align:left">If you want to play around with syntax, here is some data:</div><div style="text-align:left"><br></div><div style="text-align:left">data <- read.csv("<a href="https://raw.githubusercontent.com/rnorouzian/s/main/in.csv" target="_blank">https://raw.githubusercontent.com/rnorouzian/s/main/in.csv</a>")<br></div><div style="text-align:left"><br></div><div style="text-align:left">These combinations are encapsulated in the "study/group/outcome" piece in the following model:</div><div style="text-align:left"><br></div><div style="text-align:left">(m1 <- <a href="http://rma.mv" target="_blank">rma.mv</a>(yi, vi, random = list(~1|study/group/outcome), data = data))<br></div><div style="text-align:left"><br></div><div style="text-align:left">Now, if you force row_id onto the model above, you'll see that row_id will fill no new place here but just pushes the one already occupied by the last piece that essentially holds the GPS location of each single row i.e, .../outcome.</div><div style="text-align:left"><br></div><div style="text-align:left">(m2 <- <a href="http://rma.mv" target="_blank">rma.mv</a>(yi, vi, random = list(~1|study/group/outcome/row_id), data = data))<br></div><div style="text-align:left"><br></div><div style="text-align:left">Thus, this also means that if we remove outcome and instead use row_id we exactly reproduce m1:</div><div style="text-align:left"><br></div><div style="text-align:left">(m3 <- <a href="http://rma.mv" target="_blank">rma.mv</a>(yi, vi, random = list(~1|study/group/row_id), data = data))<br></div><div style="text-align:left"><br></div><div style="text-align:left">Kind regards,</div><div style="text-align:left">Reza</div><div style="text-align:center"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 21, 2021 at 10:51 PM Ivan Jukic <<a href="mailto:ivan.jukic@aut.ac.nz" target="_blank">ivan.jukic@aut.ac.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear Reza,<br>
<br>
Regarding the SATcoaching, I see, and since these ABC identifiers suggest these are all different studies (coming from the same authors) what you're saying makes a lot of sense. I just didn't realise that these samples are independent. Thank you!<br>
<br>
>> I don't know how your data exactly looks, but the equivalence between<br>
the two models would mean that there is not much variation in the<br>
group as a level. Therefore, in practice your second model reduces to<br>
the third model. In other words, study-group combinations = row_ids.<br>
<br>
1) I don't have groups, perhaps I misled you with the word "group" that I used in my original post. I always have a continuous moderator of interest, two outcomes, and multiple effect sizes from the same study for at least one of the two outcomes. However, I understand exactly what you mean by this. While I wanted to keep this discussion rather conceptual, I'll provide an example of my dataset just to avoid potential confusion (see at the bottom).<br>
<br>
2) By "study-group combinations" terminology to describe your row_id you just mean es_id, or number for each entry in the dataset (es_id = 1:n()), isn't it?<br>
<br>
3) Maybe I confused you with my message, what I meant by the fact that the models are the same is that they are both returning *identical* outputs (all estimates). Actually, the second model *also* returns gamma and phi estimates which are missing in the third model you described. The second model seems to be overparameterized because "Some combinations of the levels of the inner factor never occurred. Corresponding phi value(s) fixed to 0". This is not happening with the third model, of course. I guess I misspecified it due to the confusion of the data structure?<br>
<br>
>>... please take this answer as just providing some intuition/heuristics).<br>
<br>
Of course! I really appreciate this conceptual level discussion, and you provided much more than I expected, so thank you very much!<br>
<br>
study ---- mod1 ---- outcome ---- es_id<br>
1 15 1 1<br>
1 15 1 2<br>
1 15 0 3<br>
1 15 0 4<br>
1 30 1 5<br>
1 30 1 6<br>
1 30 0 7<br>
1 30 0 8<br>
.<br>
.<br>
.<br>
<br>
Cheers,<br>
Ivan<br>
<br>
----<br>
<br>
From: Reza Norouzian <<a href="mailto:rnorouzian@gmail.com" target="_blank">rnorouzian@gmail.com</a>><br>
Sent: Friday, 22 October 2021 6:47 AM<br>
To: Ivan Jukic <<a href="mailto:ivan.jukic@aut.ac.nz" target="_blank">ivan.jukic@aut.ac.nz</a>><br>
Cc: <a href="mailto:r-sig-meta-analysis@r-project.org" target="_blank">r-sig-meta-analysis@r-project.org</a> <<a href="mailto:r-sig-meta-analysis@r-project.org" target="_blank">r-sig-meta-analysis@r-project.org</a>><br>
Subject: Re: [R-meta] Multivariate meta regression and predict for robust estimates <br>
<br>
Hi Ivan,<br>
<br>
Please see my answers inline.<br>
<br>
>> With regards to the SATcoaching example, how so [how come the same level of test doesn't repeat in the same study]?<br>
<br>
Note that studies in SATcoaching data have (A), (B) suffixes. Thus,<br>
each of these is different studies. As a result, in the table of test<br>
levels (co-)occurrences below, we can see that the number of studies<br>
(outside parentheses) that contain each (Math or Verbal) or both (Math<br>
and Verbal) levels of test and the corresponding number of effect<br>
sizes/rows (in parentheses below) for each of these cases are the<br>
same:<br>
<br>
test Math Verbal<br>
1 Math 29 (29) -<br>
2 Verbal 20 (20) 38 (38)<br>
<br>
This means that we don't have the repetition of the same levels of<br>
test in the same study (this answers your next question as well).<br>
There are 20 studies in which Verbal and Math occur only once<br>
together. That's why there is no need for a further level beyond study<br>
and "random = ~ test | study" suffices.<br>
<br>
>> You mean no repetition of the same level of outcome occurs within the same sample, perhaps?<br>
<br>
See my answer above.<br>
<br>
>> 1) The second and third models should effectively be the same, and they are, after adding what was missing to the second one (~ 1 | es_id). While the syntax of the third one makes a lot of sense, I'm struggling to understand the syntax of the second one, and ultimately, why are they the same?<br>
<br>
second_model <- <a href="http://rma.mv" rel="noreferrer" target="_blank">rma.mv</a>(yi, V, random = list(~ outcome | study, ~ outcome |<br>
interaction(study, group), ~1|row_id), struct = c("UN","UN"))<br>
<br>
third_model <- <a href="http://rma.mv" rel="noreferrer" target="_blank">rma.mv</a>(yi, V, random = list(~ outcome | study, ~ 1|<br>
row_id), struct = "UN"))<br>
<br>
I don't know how your data exactly looks, but the equivalence between<br>
the two models would mean that there is not much variation in the<br>
group as a level. Therefore, in practice your second model reduces to<br>
the third model. In other words, study-group combinations = row_ids.<br>
<br>
You can roughly translate the two models, respectively, to:<br>
<br>
~ 1 | study/group/outcome/row_id<br>
<br>
~ 1 | study/outcome/row_id<br>
<br>
Now, you can see that, if these two models are the same, it's due to<br>
the group not effectively being a level.<br>
<br>
>> 2) When you say "coded for" and "haven't coded for" the design-related feature(s) you are literally referring to having vs not having all "columns" related to study, groups, and outcomes properly aligned, right? I guess it's hard for me to relate as I always have these three together with es_id (or row_id, as you say) as a fourth one.<br>
<br>
Yes e.g., meaning that studies can have various groups, outcomes,<br>
times of measurement, independent samples of subjects, . . . .<br>
<br>
Potentially, each of these features can be a column in the dataset. In<br>
practice, however, accounting for all these sources of variation may<br>
be a theoretical ideal. Because, rarely do so many studies<br>
simultaneously possess all these features. If they do, then, you would<br>
expect a super large dataset where each feature accounts for the<br>
variation within the feature that sits above it. Here is one example<br>
study where each of the features mentioned above have only two<br>
observed levels (e.g., just pre and post, just two outcomes, just<br>
two...). You can imagine what happens to the number of rows in this<br>
one study, if some of these features have more than two observed<br>
levels!<br>
<br>
So all of this means that without your exact data, RQs, goals,<br>
context, and substantive expertise regarding the current state of<br>
literature/phenomenon under study fitting these models wouldn't be<br>
possible. (So, please take this answer as just providing some<br>
intuition/heuristics).<br>
<br>
</blockquote></div>
</div>