[R-meta] [External] RE: Question about escalc, proportion ES, and nested data

[Viechtbauer, Wolfgang (SP)]

You don't have to derive an effect size - the proportions you have *are* your 'effect size' (leaving aside whether some transformation thereof may be beneficial). The problem is that you can't really compute any sensible estimate of their sampling variances without either doing an analysis that would directly produce the corresponding SEs or deriving the distribution properties of these proportions based on statistical theory. One possibility would be to just use 1/n or something like that as a very rough approximation of the sampling variances and then apply (cluster) robust inference methods.

Best,
Wolfgang

>-----Original Message-----
>From: Harris, Jordan L [mailto:jordan-l-harris using uiowa.edu]
>Sent: Wednesday, 02 February, 2022 18:48
>To: Viechtbauer, Wolfgang (SP)
>Cc: r-sig-meta-analysis using r-project.org
>Subject: Re: [External] RE: Question about escalc, proportion ES, and nested data
>
>Hi Dr. Viechtbauer (and all),
>
>Thank you very much for the reply!
>
>I modeled the data using the available standardized loadings from the study's
>factor analysis. To derive a proportion of variance in each factor I squared the
>standardized loadings and averaged them. This was done for all of the specific
>factors (e.g., internalizing, and externalizing), and the general factor (sharing
>the same indicators as all the specific factors). I summed all these values
>(specific + general) to derive a "total" variance score, from which I divided the
>general variance score to calculate my variable of interest (i.e., general /
>general + specific). Unfortunately, I do not have standard errors from this
>method as I used excel functions to calculate these scores.
>
>Given this context, do you have any recommendations for the measure and method by
>which I can derive effect size and sampling variance using escalc?
>
>It might also help to know that I have sample size "sample_n" information from
>all unique timepoints and samples. The hope is that I can retain as much
>information as possible while also accounting for having the same participants
>assessed at multiple timepoints and not double counting them.
>
>Any guidance will be appreciated!
>Best,
>Jordan
>________________________________________
>From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: Wednesday, February 2, 2022 4:17 AM
>To: Harris, Jordan L <jordan-l-harris using uiowa.edu>; r-sig-meta-analysis using r-
>project.org <r-sig-meta-analysis using r-project.org>
>Subject: [External] RE: Question about escalc, proportion ES, and nested data
>
>Dear Jordan,
>
>Please see below for my responses.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Harris, Jordan L
>>Sent: Tuesday, 01 February, 2022 22:50
>>To: r-sig-meta-analysis using r-project.org
>>Subject: [R-meta] Question about escalc, proportion ES, and nested data
>>
>>Hi List Members,
>>
>>I am a graduate student who is new to R and meta-analyses, and I have been
>>running into problems getting my code sorted out.
>>
>>I am conducting a meta-analysis to explore how the structure of psychopathology
>>changes across childhood and adolescence. My effect size of interest is
>>represented by a proportion score that is conceptualized as ratio of variance
>>accounted for by a general factor, called "general_es" (i.e., general / general
>+
>>specific). These data do not currently have a sampling variance, nor have
>>transformed effect sizes been calculated. I have 3 levels of nested data: Level
>1
>>= "timepoint_id", Level 2 = "sample_id", Level 1 = "study_id" which account for
>>non-independence of data. Here, I will call my data file "dat."
>>
>>  1.  How should I structure the escalc command to derive a "yi" and "V" values
>>needed for the rma.mv analysis? Would my measure be "PLO"?
>
>"PLO" is for binomial data, which is not what you appear to have. A logit
>transformation may in itself be useful for proportions (however derived), but the
>calculation of the sampling variance in escalc() assumes that each proportion was
>calculated based on a random variable that follows a binomial distribution.
>
>Ideally, one would need the standard errors of the proportions, which should come
>from whatever method/model was used to obtain those proportions. Then one can use
>the delta method to obtain the sampling variances of the logit-transformed
>proportions.
>
>Getting the covariance between sampling errors would be even more difficult
>(multiple proportions obtained from the same sample will have non-zero
>correlations between the sampling errors).
>
>>  2.  Would this structure be acceptable: rma.mv(yi, vi, random = ~ 1 |
>>study_id/sample_id/timepoint_id, data=dat)?
>
>Possibly, but it is impossible to answer this properly without further details.
>For example, this model assumes constant correlation across timepoints,
>regardless of how far they are apart.
>
>And as noted above, this model would not account for non-independent sampling
>errors.
>
>>Thanks,
>>Jordan