[R-meta] escale ROM or SMD

[Lukas Dylewski]

Hi James,

thank you for your advice. They were very helpful !
I will ask other coauthors who work with environmental stress, however, we
want to predict effect size for a tree type with average mass and for a
drought of average length.

Best

L



czw., 15 lip 2021, 19:24 użytkownik James Pustejovsky <jepusto using gmail.com>
napisał:

> Hi Lukasz,
>
> This depends on what exactly you mean by "mean effect size." Your code
> gives you the predicted effect size for a coniferous tree with average mass
> (average across all types of trees) and for a drought of average length.
>
> But perhaps you are trying to estimate the average effect size across the
> population of trees and drought events. If so, then I think you would need
> to use the average of the squared terms (which is not identical to the
> square of the average terms). Example code as follows:
>
> m <- mean(hedges$logmass)
> m_sq <-  mean(hedges$logmass^2)
> n <- mean(hedges$drougth.day)
> n_sq <- mean(hedges$drougth.day^2)
>
> For conifeorus:
> predict(res1, newmods = c(m, m_sq, 0, n, n_sq))
>
> For deciduous:
> predict(res1, newmods = c(m, m_sq, 1,n,n_sq))
>
> Or perhaps you are trying to estimate something else, like the average
> effect sizes within the sub-populations of deciduous trees and coniferous
> trees (which might be different because deciduous trees are larger or
> smaller than coniferous trees, or grow in regions more or less prone to
> droughts).
>
> Can you say more about the scientific question you are trying to answer by
> calculating mean effect sizes?
>
> James
>
> On Wed, Jul 14, 2021 at 2:59 PM Lukas Dylewski <dylewski91 using gmail.com>
> wrote:
>
>> Hi James,
>>
>> thank you very much for your help!
>>
>> I checked and I think that proline is measure as a ratio scale. The
>> proline unit is given as μmoles proline / g of the fresh or dry weight of
>> plant material.
>>
>> I have another technical question
>> I would like to calculate the mean effect size for one of the category
>> moderator: plant type (coniferous and deciduous separately) based on the
>> full model including all fixed effects. Does this code correct?
>>
>> m <- mean(hedges$logmass)
>> n <- mean(hedges$drougth.day)
>>
>> For conifeorus:
>> predict(res1, newmods = c(m, m^2, 0,n,n^2))
>>
>> For deciduous:
>> predict(res1, newmods = c(m, m^2, 1,n,n^2))
>>
>> Once again thank you for your help I  appreciate it.
>>
>> Best
>> Lukasz
>>
>>
>> wt., 13 lip 2021, 23:25 użytkownik James Pustejovsky <jepusto using gmail.com>
>> napisał:
>>
>>> HI Lukasz,
>>>
>>> Regarding question 1 about measurement, I was wondering about whether
>>> proline concentration is on a ratio scale, meaning a scale where ratio
>>> comparisons are meaningful (see eg
>>> https://en.wikipedia.org/wiki/Level_of_measurement#Ratio_scale). Does a
>>> proline concentration of zero mean that there is no proline accumulation at
>>> all? What are the *units* of proline concentration measurements?
>>>
>>> Regarding question 3 about empirical features, I see from Figure 1 that
>>> there seems to be a relationship between the M and the SD---particularly in
>>> control conditions. That would suggest that ROM might be more appropriate
>>> than SMD.
>>>
>>> Regarding question 4, heterogeneity, I see in Figure 2 that there is a
>>> very strong relationship between the ES and its sampling variance when
>>> using the SMD metric. That relationship is artificial (see
>>> https://doi.org/10.1002/jrsm.1332). I also see that there is a really
>>> extreme degree of variation in the SMD effect size estimates. Values of 20+
>>> for a SMD are just non-sensical, in my view. In contrast, the ROM estimates
>>> vary over a wide range, but not an absurd one, and their sampling variances
>>> are much more homogeneous (with the exception of the one outlier). All of
>>> this further supports the use of ROM, I think.
>>>
>>> James
>>>
>>> On Mon, Jul 12, 2021 at 12:45 PM Lukas Dylewski <dylewski91 using gmail.com>
>>> wrote:
>>>
>>>> Dear James,
>>>> thank you for quick response !
>>>>
>>>> Here is my response:
>>>> *1.  Measurement properties: what is proline concentration? Is it
>>>> measured on a ratio scale, such that ROM is a sensible way to describe
>>>> change (over time or as a result of intervention)? If you provide a bit
>>>> more detail about what the outcome is (for us social scientists on the
>>>> listserv), then perhaps others will be able to weigh in also. *
>>>> Response: Proline concentration is measure as some value not ratio
>>>> (e.g. mmol g-1 DM). For example in the control group and experimental group
>>>> (drought stress) I have some proline value in plant tissue.  In this
>>>> research, we want to check the overall effect of drought stress on proline
>>>> concentration, and how the duration of drought (in days) affected proline
>>>> concentration. Moreover, proline concentration, not a significant change in
>>>> well water treatment during the time. So in the publication, they give the
>>>> value not the ratio of change proline during the duration of drought. So,
>>>> when in one publication authors provide results for proline concentration
>>>> during the duration of drought (e.g. control, 1-day drought, 7 days
>>>> drought and 14 days drought, the effect size I calculate based on control
>>>> group compared with the drought day group, the effect size for 1 day is
>>>> calculated based on control (mean/SD/n) vs. 1 day (mean/SD/n); effect size
>>>> for 7 days is control vs. 7 days; effect size for 14 days is control vs. 14
>>>> days.
>>>>
>>>> 2. *Theory: Is there any relevant botanical theory that would indicate
>>>> how drought stress should be related to proline concentration? *
>>>> Response: The phenomenon of proline accumulation in plant tissue is
>>>> known to occur under environmental stress water deficit, salinity, low
>>>> temperature, etc. So the effect should be also positive on proline
>>>> concentration activate by some stressor. However, there are no studies
>>>> showing how strong this effect is for e.g. the type of plants, duration of
>>>> drought, or seed size. We know that in most studies water stress has a
>>>> positive effect on proline concentration.
>>>>
>>>> 3. *Empirical features: It can be helpful to create scatterplots
>>>> showing the relationship between the M and the SD of the outcome in each
>>>> group and between the Ms in different comparison groups. If the outcomes
>>>> are on drastically different scales, then the plots can be created within
>>>> subgroups that use the same or similar measurement instruments. If the SD
>>>> of the outcome is strongly related to the M, then I would take this as an
>>>> indication that ROM might be more appropriate than SMD.*
>>>> Response: I attach the graph, I hope I understood correctly. Fig. 1 is
>>>> a scatter plot of mean and SD for each group control and experiment. Fig. 2
>>>> is a scatter plot for effect size (yi) to vi in both cases ROM and SMD.
>>>>
>>>> *4) Heterogeneity: Which model has less unexplained variability (as
>>>> measured by I^2, for instance)? All else equal, I would prefer the effect
>>>> size metric where the meta-analytic model has greater explanatory power.
>>>> From what you've said, it sounds like the ROM would win out here due to
>>>> fewer outliers and no apparent funnel plot asymmetry.*
>>>> Response:
>>>> res0 <- rma(yi, vi,method="REML",data=hedges) - ROM
>>>> For ROM - I^2 equal 99.76%; effect size 0.6715
>>>>
>>>> res0 <- rma(yi, vi,method="REML",data=hedges2) - SMD
>>>> For SMD - I^2 equal 99.23%; effect size 7.6561
>>>>
>>>> Thank you for help !
>>>>
>>>> Best
>>>>
>>>> Lukasz
>>>>
>>>> pon., 12 lip 2021 o 16:10 James Pustejovsky <jepusto using gmail.com>
>>>> napisał(a):
>>>>
>>>>> Hi Lukas,
>>>>>
>>>>> I think there are (at least) four relevant considerations here:
>>>>> 1) Measurement properties: what is proline concentration? Is it
>>>>> measured on a ratio scale, such that ROM is a sensible way to describe
>>>>> change (over time or as a result of intervention)? If you provide a bit
>>>>> more detail about what the outcome is (for us social scientists on the
>>>>> listserv), then perhaps others will be able to weigh in also.
>>>>> 2) Theory: Is there any relevant botanical theory that would indicate
>>>>> how drought stress should be related to proline concentration?
>>>>> 3) Empirical features: It can be helpful to create scatterplots
>>>>> showing the relationship between the M and the SD of the outcome in each
>>>>> group and between the Ms in different comparison groups. If the outcomes
>>>>> are on drastically different scales, then the plots can be created within
>>>>> subgroups that use the same or similar measurement instruments. If the SD
>>>>> of the outcome is strongly related to the M, then I would take this as an
>>>>> indication that ROM might be more appropriate than SMD.
>>>>> 4) Heterogeneity: Which model has less unexplained variability (as
>>>>> measured by I^2, for instance)? All else equal, I would prefer the effect
>>>>> size metric where the meta-analytic model has greater explanatory power.
>>>>> From what you've said, it sounds like the ROM would win out here due to
>>>>> fewer outliers and no apparent funnel plot asymmetry.
>>>>>
>>>>> I've listed these considerations in what I would consider to be
>>>>> decreasing order of priority (1st being essential, 2nd being important, 3rd
>>>>> and 4th being matters of judgement). Others might have different
>>>>> perspectives, though.
>>>>>
>>>>> Kind Regards,
>>>>> James
>>>>>
>>>>> On Mon, Jul 12, 2021 at 5:43 AM Lukas Dylewski <dylewski91 using gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear All,
>>>>>>
>>>>>> I conducted a mixed meta-regression model to check the effect of
>>>>>> drought stress on proline concentration in leaves. In the model, I include
>>>>>> the following moderators: duration of drought (in the day), seed mass, and
>>>>>> plant type (coniferous vs. deciduous). The duration of the drought affects
>>>>>> the proline content, therefore it was included in the model.
>>>>>>
>>>>>> When I calculate Hedge g (measure=SMD) values I got very big values
>>>>>> for some records (e.g. 100, 50, etc.) and publication bias. However, when I
>>>>>> calculate effect size using ROM I got very nice numbers without publication
>>>>>> bias.
>>>>>>
>>>>>> My first question: Can I use the ROM method for my dataset (I attach
>>>>>> the database: proline) or should I use the SMD?
>>>>>>
>>>>>> This is my code (with ROM measure)
>>>>>>
>>>>>> z<-proline
>>>>>> z$logmass<-log(z$seed.mass.mg+1)
>>>>>>
>>>>>> hedges<-escalc(measure="ROM",data=z,append=T,m1i=experiment,n1i=n.experiment,sd1i=sd.experiment,m2i=control,n2i=n.control,sd2i=sd.control)
>>>>>> hedges
>>>>>>
>>>>>> res1 <- rma.mv(yi, vi, mods = ~ logmass + I(logmass^2)+
>>>>>> factor(plant.type) + drougth.day + I(drougth.day^2),
>>>>>> random=~1|references,data=hedges, method="REML")
>>>>>>
>>>>>> After the model summary, I would like to calculate the mean effect
>>>>>> size for the plant type (coniferous and deciduous separately). Does this
>>>>>> code correct?
>>>>>>
>>>>>> m <- mean(hedges$logmass)
>>>>>> n <- mean(hedges$drougth.day)
>>>>>>
>>>>>> For conifeorus:
>>>>>> predict(res1, newmods = c(m, m^2, 0,n,n^2))
>>>>>>
>>>>>> For deciduous:
>>>>>> predict(res1, newmods = c(m, m^2, 1,n,n^2))
>>>>>>
>>>>>> --
>>>>>> Łukasz Dylewski, PhD.
>>>>>>
>>>>>> Institute of Dendrology,
>>>>>>
>>>>>> Polish Academy of Sciences,
>>>>>>
>>>>>> Parkowa 5, 62-035 Kórnik, Poland
>>>>>> _______________________________________________
>>>>>> R-sig-meta-analysis mailing list
>>>>>> R-sig-meta-analysis using r-project.org
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Łukasz Dylewski, PhD.
>>>>
>>>> Institute of Dendrology,
>>>>
>>>> Polish Academy of Sciences,
>>>>
>>>> Parkowa 5, 62-035 Kórnik, Poland
>>>>
>>>

	[[alternative HTML version deleted]]