[R-meta] escale ROM or SMD

[James Pustejovsky]

HI Lukasz,

Regarding question 1 about measurement, I was wondering about whether
proline concentration is on a ratio scale, meaning a scale where ratio
comparisons are meaningful (see eg
https://en.wikipedia.org/wiki/Level_of_measurement#Ratio_scale). Does a
proline concentration of zero mean that there is no proline accumulation at
all? What are the *units* of proline concentration measurements?

Regarding question 3 about empirical features, I see from Figure 1 that
there seems to be a relationship between the M and the SD---particularly in
control conditions. That would suggest that ROM might be more appropriate
than SMD.

Regarding question 4, heterogeneity, I see in Figure 2 that there is a very
strong relationship between the ES and its sampling variance when using the
SMD metric. That relationship is artificial (see
https://doi.org/10.1002/jrsm.1332). I also see that there is a really
extreme degree of variation in the SMD effect size estimates. Values of 20+
for a SMD are just non-sensical, in my view. In contrast, the ROM estimates
vary over a wide range, but not an absurd one, and their sampling variances
are much more homogeneous (with the exception of the one outlier). All of
this further supports the use of ROM, I think.

James

On Mon, Jul 12, 2021 at 12:45 PM Lukas Dylewski <dylewski91 using gmail.com>
wrote:

> Dear James,
> thank you for quick response !
>
> Here is my response:
> *1.  Measurement properties: what is proline concentration? Is it measured
> on a ratio scale, such that ROM is a sensible way to describe change (over
> time or as a result of intervention)? If you provide a bit more detail
> about what the outcome is (for us social scientists on the listserv), then
> perhaps others will be able to weigh in also. *
> Response: Proline concentration is measure as some value not ratio (e.g.
> mmol g-1 DM). For example in the control group and experimental group
> (drought stress) I have some proline value in plant tissue.  In this
> research, we want to check the overall effect of drought stress on proline
> concentration, and how the duration of drought (in days) affected proline
> concentration. Moreover, proline concentration, not a significant change in
> well water treatment during the time. So in the publication, they give the
> value not the ratio of change proline during the duration of drought. So,
> when in one publication authors provide results for proline concentration
> during the duration of drought (e.g. control, 1-day drought, 7 days
> drought and 14 days drought, the effect size I calculate based on control
> group compared with the drought day group, the effect size for 1 day is
> calculated based on control (mean/SD/n) vs. 1 day (mean/SD/n); effect size
> for 7 days is control vs. 7 days; effect size for 14 days is control vs. 14
> days.
>
> 2. *Theory: Is there any relevant botanical theory that would indicate
> how drought stress should be related to proline concentration? *
> Response: The phenomenon of proline accumulation in plant tissue is known
> to occur under environmental stress water deficit, salinity, low
> temperature, etc. So the effect should be also positive on proline
> concentration activate by some stressor. However, there are no studies
> showing how strong this effect is for e.g. the type of plants, duration of
> drought, or seed size. We know that in most studies water stress has a
> positive effect on proline concentration.
>
> 3. *Empirical features: It can be helpful to create scatterplots showing
> the relationship between the M and the SD of the outcome in each group and
> between the Ms in different comparison groups. If the outcomes are on
> drastically different scales, then the plots can be created within
> subgroups that use the same or similar measurement instruments. If the SD
> of the outcome is strongly related to the M, then I would take this as an
> indication that ROM might be more appropriate than SMD.*
> Response: I attach the graph, I hope I understood correctly. Fig. 1 is a
> scatter plot of mean and SD for each group control and experiment. Fig. 2
> is a scatter plot for effect size (yi) to vi in both cases ROM and SMD.
>
> *4) Heterogeneity: Which model has less unexplained variability (as
> measured by I^2, for instance)? All else equal, I would prefer the effect
> size metric where the meta-analytic model has greater explanatory power.
> From what you've said, it sounds like the ROM would win out here due to
> fewer outliers and no apparent funnel plot asymmetry.*
> Response:
> res0 <- rma(yi, vi,method="REML",data=hedges) - ROM
> For ROM - I^2 equal 99.76%; effect size 0.6715
>
> res0 <- rma(yi, vi,method="REML",data=hedges2) - SMD
> For SMD - I^2 equal 99.23%; effect size 7.6561
>
> Thank you for help !
>
> Best
>
> Lukasz
>
> pon., 12 lip 2021 o 16:10 James Pustejovsky <jepusto using gmail.com>
> napisał(a):
>
>> Hi Lukas,
>>
>> I think there are (at least) four relevant considerations here:
>> 1) Measurement properties: what is proline concentration? Is it measured
>> on a ratio scale, such that ROM is a sensible way to describe change (over
>> time or as a result of intervention)? If you provide a bit more detail
>> about what the outcome is (for us social scientists on the listserv), then
>> perhaps others will be able to weigh in also.
>> 2) Theory: Is there any relevant botanical theory that would indicate how
>> drought stress should be related to proline concentration?
>> 3) Empirical features: It can be helpful to create scatterplots showing
>> the relationship between the M and the SD of the outcome in each group and
>> between the Ms in different comparison groups. If the outcomes are on
>> drastically different scales, then the plots can be created within
>> subgroups that use the same or similar measurement instruments. If the SD
>> of the outcome is strongly related to the M, then I would take this as an
>> indication that ROM might be more appropriate than SMD.
>> 4) Heterogeneity: Which model has less unexplained variability (as
>> measured by I^2, for instance)? All else equal, I would prefer the effect
>> size metric where the meta-analytic model has greater explanatory power.
>> From what you've said, it sounds like the ROM would win out here due to
>> fewer outliers and no apparent funnel plot asymmetry.
>>
>> I've listed these considerations in what I would consider to be
>> decreasing order of priority (1st being essential, 2nd being important, 3rd
>> and 4th being matters of judgement). Others might have different
>> perspectives, though.
>>
>> Kind Regards,
>> James
>>
>> On Mon, Jul 12, 2021 at 5:43 AM Lukas Dylewski <dylewski91 using gmail.com>
>> wrote:
>>
>>> Dear All,
>>>
>>> I conducted a mixed meta-regression model to check the effect of drought
>>> stress on proline concentration in leaves. In the model, I include the
>>> following moderators: duration of drought (in the day), seed mass, and
>>> plant type (coniferous vs. deciduous). The duration of the drought affects
>>> the proline content, therefore it was included in the model.
>>>
>>> When I calculate Hedge g (measure=SMD) values I got very big values for
>>> some records (e.g. 100, 50, etc.) and publication bias. However, when I
>>> calculate effect size using ROM I got very nice numbers without publication
>>> bias.
>>>
>>> My first question: Can I use the ROM method for my dataset (I attach the
>>> database: proline) or should I use the SMD?
>>>
>>> This is my code (with ROM measure)
>>>
>>> z<-proline
>>> z$logmass<-log(z$seed.mass.mg+1)
>>>
>>> hedges<-escalc(measure="ROM",data=z,append=T,m1i=experiment,n1i=n.experiment,sd1i=sd.experiment,m2i=control,n2i=n.control,sd2i=sd.control)
>>> hedges
>>>
>>> res1 <- rma.mv(yi, vi, mods = ~ logmass + I(logmass^2)+
>>> factor(plant.type) + drougth.day + I(drougth.day^2),
>>> random=~1|references,data=hedges, method="REML")
>>>
>>> After the model summary, I would like to calculate the mean effect size
>>> for the plant type (coniferous and deciduous separately). Does this code
>>> correct?
>>>
>>> m <- mean(hedges$logmass)
>>> n <- mean(hedges$drougth.day)
>>>
>>> For conifeorus:
>>> predict(res1, newmods = c(m, m^2, 0,n,n^2))
>>>
>>> For deciduous:
>>> predict(res1, newmods = c(m, m^2, 1,n,n^2))
>>>
>>> --
>>> Łukasz Dylewski, PhD.
>>>
>>> Institute of Dendrology,
>>>
>>> Polish Academy of Sciences,
>>>
>>> Parkowa 5, 62-035 Kórnik, Poland
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>
> --
> Łukasz Dylewski, PhD.
>
> Institute of Dendrology,
>
> Polish Academy of Sciences,
>
> Parkowa 5, 62-035 Kórnik, Poland
>

	[[alternative HTML version deleted]]