[R-meta] Pseudoreplication

[Viechtbauer, Wolfgang (SP)]

Dear Gladys,

Along the lines of what Michael already said -- it seems like the reviewer might not be familiar with modern meta-analytic methods (like the multilevel/multivariate models we frequently discuss on this mailing list). I cannot tell you whether what you have done is fully sufficient to capture all potential sources of heterogeneity and dependency in your data, but it's a fairly standard (four-level) multilevel random-effects model that allows for correlation in multiple effects belonging to the same study and multiple studies belonging to the same article. So this does take 'pseudoreplication' into consideration.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Gladys Barragan-Jason
>Sent: Tuesday, 28 September, 2021 16:08
>To: Michael Dewey
>Cc: R meta
>Subject: Re: [R-meta] Pseudoreplication
>
>Dear Michael,
>thanks a lot for your reply.
>I have a structure like this:
>
>Article ID.         Study ID.           Effect size.
>X et al.              S1                       0.5
>X et al.              S2                       0.8
>Y et al.              S1                       0.2
>Y et al.              S2                       0.6
>Y et al.              S2                       0.2
>Z et al.              S1                       0.1
>Z et al.              S1                       0.5
>
>and I use: random=~1|articleID/studyID/estID with estid <- 1:nrow(data)
>Thanks,
>Gladys
>
>Le mar. 28 sept. 2021 à 15:57, Michael Dewey <lists using dewey.myzen.co.uk> a écrit :
>Dear Gladys
>
>I think the referee may have misunderstood what you did. We may need
>more details about your data structure to be absolutely sure you have
>the right structure but it seems superior to the suggested rather
>indirect way of going about things.
>
>Micael
>
>On 28/09/2021 13:06, Gladys Barragan-Jason wrote:
>> Dear all,
>>
>> I am contacting you to have your advice about a reviewer comment on a
>> meta-analysis. The reviewer said that as we used multiple comparisons
>> from within studies (e.g., five effect sizes from one particular paper
>> comparisons), this raises the potential of pseudoreplication issue and
>> he/she suggests to do the following:  recalculate the overall effect
>> sizes after sampling one comparison from each separate study and
>> calculate the estimated mean and 95% CI of effect size by bootstrap
>> resampling 1,000 times in R.
>> However, I think I've already taken into account non-independence of the
>> data by including three random effects using metafor to control for
>> multiple data from the same article, multiple data from same
>> participants from the same study and multiple estimates within a study
>> within a lab. So I coded random effect as follows:
>> ~1|articleID/studyID/estID. I think this is the right way to take into
>> account heterogeneity of the data but I would like to know if I am
>> correct or if the pseudoreplication techniques gave something different
>> and/or complementary?
>> Thanks in advance for your help!
>> Best,
>> Gladys