[R-meta] escale ROM or SMD

[James Pustejovsky]

Hi Lukasz,

This depends on what exactly you mean by "mean effect size." Your code
gives you the predicted effect size for a coniferous tree with average mass
(average across all types of trees) and for a drought of average length.

But perhaps you are trying to estimate the average effect size across the
population of trees and drought events. If so, then I think you would need
to use the average of the squared terms (which is not identical to the
square of the average terms). Example code as follows:

m <- mean(hedges$logmass)
m_sq <-  mean(hedges$logmass^2)
n <- mean(hedges$drougth.day)
n_sq <- mean(hedges$drougth.day^2)

For conifeorus:
predict(res1, newmods = c(m, m_sq, 0, n, n_sq))

For deciduous:
predict(res1, newmods = c(m, m_sq, 1,n,n_sq))

Or perhaps you are trying to estimate something else, like the average
effect sizes within the sub-populations of deciduous trees and coniferous
trees (which might be different because deciduous trees are larger or
smaller than coniferous trees, or grow in regions more or less prone to
droughts).

Can you say more about the scientific question you are trying to answer by
calculating mean effect sizes?

James

On Wed, Jul 14, 2021 at 2:59 PM Lukas Dylewski <dylewski91 using gmail.com> wrote:

> Hi James,
>
> thank you very much for your help!
>
> I checked and I think that proline is measure as a ratio scale. The
> proline unit is given as μmoles proline / g of the fresh or dry weight of
> plant material.
>
> I have another technical question
> I would like to calculate the mean effect size for one of the category
> moderator: plant type (coniferous and deciduous separately) based on the
> full model including all fixed effects. Does this code correct?
>
> m <- mean(hedges$logmass)
> n <- mean(hedges$drougth.day)
>
> For conifeorus:
> predict(res1, newmods = c(m, m^2, 0,n,n^2))
>
> For deciduous:
> predict(res1, newmods = c(m, m^2, 1,n,n^2))
>
> Once again thank you for your help I  appreciate it.
>
> Best
> Lukasz
>
>
> wt., 13 lip 2021, 23:25 użytkownik James Pustejovsky <jepusto using gmail.com>
> napisał:
>
>> HI Lukasz,
>>
>> Regarding question 1 about measurement, I was wondering about whether
>> proline concentration is on a ratio scale, meaning a scale where ratio
>> comparisons are meaningful (see eg
>> https://en.wikipedia.org/wiki/Level_of_measurement#Ratio_scale). Does a
>> proline concentration of zero mean that there is no proline accumulation at
>> all? What are the *units* of proline concentration measurements?
>>
>> Regarding question 3 about empirical features, I see from Figure 1 that
>> there seems to be a relationship between the M and the SD---particularly in
>> control conditions. That would suggest that ROM might be more appropriate
>> than SMD.
>>
>> Regarding question 4, heterogeneity, I see in Figure 2 that there is a
>> very strong relationship between the ES and its sampling variance when
>> using the SMD metric. That relationship is artificial (see
>> https://doi.org/10.1002/jrsm.1332). I also see that there is a really
>> extreme degree of variation in the SMD effect size estimates. Values of 20+
>> for a SMD are just non-sensical, in my view. In contrast, the ROM estimates
>> vary over a wide range, but not an absurd one, and their sampling variances
>> are much more homogeneous (with the exception of the one outlier). All of
>> this further supports the use of ROM, I think.
>>
>> James
>>
>> On Mon, Jul 12, 2021 at 12:45 PM Lukas Dylewski <dylewski91 using gmail.com>
>> wrote:
>>
>>> Dear James,
>>> thank you for quick response !
>>>
>>> Here is my response:
>>> *1.  Measurement properties: what is proline concentration? Is it
>>> measured on a ratio scale, such that ROM is a sensible way to describe
>>> change (over time or as a result of intervention)? If you provide a bit
>>> more detail about what the outcome is (for us social scientists on the
>>> listserv), then perhaps others will be able to weigh in also. *
>>> Response: Proline concentration is measure as some value not ratio (e.g.
>>> mmol g-1 DM). For example in the control group and experimental group
>>> (drought stress) I have some proline value in plant tissue.  In this
>>> research, we want to check the overall effect of drought stress on proline
>>> concentration, and how the duration of drought (in days) affected proline
>>> concentration. Moreover, proline concentration, not a significant change in
>>> well water treatment during the time. So in the publication, they give the
>>> value not the ratio of change proline during the duration of drought. So,
>>> when in one publication authors provide results for proline concentration
>>> during the duration of drought (e.g. control, 1-day drought, 7 days
>>> drought and 14 days drought, the effect size I calculate based on control
>>> group compared with the drought day group, the effect size for 1 day is
>>> calculated based on control (mean/SD/n) vs. 1 day (mean/SD/n); effect size
>>> for 7 days is control vs. 7 days; effect size for 14 days is control vs. 14
>>> days.
>>>
>>> 2. *Theory: Is there any relevant botanical theory that would indicate
>>> how drought stress should be related to proline concentration? *
>>> Response: The phenomenon of proline accumulation in plant tissue is
>>> known to occur under environmental stress water deficit, salinity, low
>>> temperature, etc. So the effect should be also positive on proline
>>> concentration activate by some stressor. However, there are no studies
>>> showing how strong this effect is for e.g. the type of plants, duration of
>>> drought, or seed size. We know that in most studies water stress has a
>>> positive effect on proline concentration.
>>>
>>> 3. *Empirical features: It can be helpful to create scatterplots
>>> showing the relationship between the M and the SD of the outcome in each
>>> group and between the Ms in different comparison groups. If the outcomes
>>> are on drastically different scales, then the plots can be created within
>>> subgroups that use the same or similar measurement instruments. If the SD
>>> of the outcome is strongly related to the M, then I would take this as an
>>> indication that ROM might be more appropriate than SMD.*
>>> Response: I attach the graph, I hope I understood correctly. Fig. 1 is a
>>> scatter plot of mean and SD for each group control and experiment. Fig. 2
>>> is a scatter plot for effect size (yi) to vi in both cases ROM and SMD.
>>>
>>> *4) Heterogeneity: Which model has less unexplained variability (as
>>> measured by I^2, for instance)? All else equal, I would prefer the effect
>>> size metric where the meta-analytic model has greater explanatory power.
>>> From what you've said, it sounds like the ROM would win out here due to
>>> fewer outliers and no apparent funnel plot asymmetry.*
>>> Response:
>>> res0 <- rma(yi, vi,method="REML",data=hedges) - ROM
>>> For ROM - I^2 equal 99.76%; effect size 0.6715
>>>
>>> res0 <- rma(yi, vi,method="REML",data=hedges2) - SMD
>>> For SMD - I^2 equal 99.23%; effect size 7.6561
>>>
>>> Thank you for help !
>>>
>>> Best
>>>
>>> Lukasz
>>>
>>> pon., 12 lip 2021 o 16:10 James Pustejovsky <jepusto using gmail.com>
>>> napisał(a):
>>>
>>>> Hi Lukas,
>>>>
>>>> I think there are (at least) four relevant considerations here:
>>>> 1) Measurement properties: what is proline concentration? Is it
>>>> measured on a ratio scale, such that ROM is a sensible way to describe
>>>> change (over time or as a result of intervention)? If you provide a bit
>>>> more detail about what the outcome is (for us social scientists on the
>>>> listserv), then perhaps others will be able to weigh in also.
>>>> 2) Theory: Is there any relevant botanical theory that would indicate
>>>> how drought stress should be related to proline concentration?
>>>> 3) Empirical features: It can be helpful to create scatterplots showing
>>>> the relationship between the M and the SD of the outcome in each group and
>>>> between the Ms in different comparison groups. If the outcomes are on
>>>> drastically different scales, then the plots can be created within
>>>> subgroups that use the same or similar measurement instruments. If the SD
>>>> of the outcome is strongly related to the M, then I would take this as an
>>>> indication that ROM might be more appropriate than SMD.
>>>> 4) Heterogeneity: Which model has less unexplained variability (as
>>>> measured by I^2, for instance)? All else equal, I would prefer the effect
>>>> size metric where the meta-analytic model has greater explanatory power.
>>>> From what you've said, it sounds like the ROM would win out here due to
>>>> fewer outliers and no apparent funnel plot asymmetry.
>>>>
>>>> I've listed these considerations in what I would consider to be
>>>> decreasing order of priority (1st being essential, 2nd being important, 3rd
>>>> and 4th being matters of judgement). Others might have different
>>>> perspectives, though.
>>>>
>>>> Kind Regards,
>>>> James
>>>>
>>>> On Mon, Jul 12, 2021 at 5:43 AM Lukas Dylewski <dylewski91 using gmail.com>
>>>> wrote:
>>>>
>>>>> Dear All,
>>>>>
>>>>> I conducted a mixed meta-regression model to check the effect of
>>>>> drought stress on proline concentration in leaves. In the model, I include
>>>>> the following moderators: duration of drought (in the day), seed mass, and
>>>>> plant type (coniferous vs. deciduous). The duration of the drought affects
>>>>> the proline content, therefore it was included in the model.
>>>>>
>>>>> When I calculate Hedge g (measure=SMD) values I got very big values
>>>>> for some records (e.g. 100, 50, etc.) and publication bias. However, when I
>>>>> calculate effect size using ROM I got very nice numbers without publication
>>>>> bias.
>>>>>
>>>>> My first question: Can I use the ROM method for my dataset (I attach
>>>>> the database: proline) or should I use the SMD?
>>>>>
>>>>> This is my code (with ROM measure)
>>>>>
>>>>> z<-proline
>>>>> z$logmass<-log(z$seed.mass.mg+1)
>>>>>
>>>>> hedges<-escalc(measure="ROM",data=z,append=T,m1i=experiment,n1i=n.experiment,sd1i=sd.experiment,m2i=control,n2i=n.control,sd2i=sd.control)
>>>>> hedges
>>>>>
>>>>> res1 <- rma.mv(yi, vi, mods = ~ logmass + I(logmass^2)+
>>>>> factor(plant.type) + drougth.day + I(drougth.day^2),
>>>>> random=~1|references,data=hedges, method="REML")
>>>>>
>>>>> After the model summary, I would like to calculate the mean effect
>>>>> size for the plant type (coniferous and deciduous separately). Does this
>>>>> code correct?
>>>>>
>>>>> m <- mean(hedges$logmass)
>>>>> n <- mean(hedges$drougth.day)
>>>>>
>>>>> For conifeorus:
>>>>> predict(res1, newmods = c(m, m^2, 0,n,n^2))
>>>>>
>>>>> For deciduous:
>>>>> predict(res1, newmods = c(m, m^2, 1,n,n^2))
>>>>>
>>>>> --
>>>>> Łukasz Dylewski, PhD.
>>>>>
>>>>> Institute of Dendrology,
>>>>>
>>>>> Polish Academy of Sciences,
>>>>>
>>>>> Parkowa 5, 62-035 Kórnik, Poland
>>>>> _______________________________________________
>>>>> R-sig-meta-analysis mailing list
>>>>> R-sig-meta-analysis using r-project.org
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>
>>>
>>> --
>>> Łukasz Dylewski, PhD.
>>>
>>> Institute of Dendrology,
>>>
>>> Polish Academy of Sciences,
>>>
>>> Parkowa 5, 62-035 Kórnik, Poland
>>>
>>

	[[alternative HTML version deleted]]