[R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

Fri Jun 15 07:20:58 CEST 2018

Thanks all,

My original question appears to be now two (1) the distribution of my DV (hence what models to use; (2) specification of my lmer model to parse out variance components.

Topic (1): DV distribution

Yes, my measure is a sliding rule between 1-10 of subjective pain, so any number up to a single decimal is plausible. Is a linear model automatically excluded, or can (a) do a fitted/residual plot for checking; (b) log transform the dv if (a) shows evidence of  non-normality.

Going back to Rolf's point of social science, you are right. But realistically, many measures in biomechanics (which I am in), are analyzed using linear models, even though they are bounded. Example, a simple scalar height is bounded to a lower limit of zero, and an upper limit of what ever instrument is created. Joint angles are bounded physiologically too. So when are measures really -inf/inf?

Topic (2): lmer

Assuming my DV is appropriate for lmer, base on the experimental design used, I hope to receive some feedback on my fixed and random effects specification still 😊

Thanks again all, for the kind response

Bernard

-----Original Message-----
From: bbolker using gmail.com <bbolker using gmail.com>
Sent: Friday, June 15, 2018 2:28 AM
To: Doran, Harold <HDoran using air.org>
Cc: Rolf Turner <r.turner using auckland.ac.nz>; r-sig-mixed-models using r-project.org; Bernard Liew <B.Liew using bham.ac.uk>
Subject: Re: [R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

More generally, the best way to fit this kind of model is to use an

*ordinal* model, which assumes the responses are in increasing sequence but does not assume the distance between levels (e.g. 1 vs 2,

2 vs 3 ...) is uniform.  However, I'm not sure how one would go about computing an ICC from ordinal data ... (the 'ordinal' package is the place to look for the model-fitting procedures). Googling it finds some stuff, but it seems that it doesn't necessarily apply to complex designs ...

https://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/

On Thu, Jun 14, 2018 at 6:58 PM, Doran, Harold <HDoran using air.org<mailto:HDoran using air.org>> wrote:

> That’s a helpful clarification, Rolf. However, with gaussian normal

> errors in the linear model, we can’t *really* assume they would

> asymptote at 1 or 10. My suspicion is that these are likert-style

> ordered counts of some form, although the OP should clarify. In which

> case, the 1 or 10 are limits with censoring, as true values for some

> measured trait could exist outside those boundaries (and I suspect the

> model is forming predicted values outside of 1 or 10).

>

>

>

> On 6/14/18, 6:33 PM, "Rolf Turner" <r.turner using auckland.ac.nz<mailto:r.turner using auckland.ac.nz>> wrote:

>

>>

>>On 15/06/18 05:35, Doran, Harold wrote:

>>

>>> Well no, you¹re specification is not right because your variable is

>>> not continuous as you note. Continuous means it is a real number

>>> between -Inf/Inf and you have boundaries between 1 and 10. So, you

>>> should not be using a linear model assuming the outcome is continuous.

>>

>>I think that the foregoing is a bit misleading.  For a variable to be

>>continuous it is not necessary for it to have a range from -infinity

>>to infinity.

>>

>>The OP says that dv  "is a continuous variable (scale 1-10)".  It is

>>not clear to me what this means.  The "obvious"/usual meaning or

>>interpretation would be that dv can take (only) the (positive integer)

>>values 1, 2, ..., 10.  If this is so, then a continuous model is not

>>appropriate.  (It should be noted however that people in the social

>>sciences do this sort of thing --- i.e. treat discrete variables as

>>continuous --- all the time.)

>>

>>It is *possible* that dv can take values in the real interval [1,10],

>>in which case it *is* continuous, and a "continuous model" is indeed

>>appropriate.

>>

>>The OP should clarify what the situation actually is.

>>

>>cheers,

>>

>>Rolf Turner

>>

>>--

>>Technical Editor ANZJS

>>Department of Statistics

>>University of Auckland

>>Phone: +64-9-373-7599 ext. 88276

>>

>>> On 6/14/18, 11:16 AM, "Bernard Liew" <B.Liew using bham.ac.uk<mailto:B.Liew using bham.ac.uk>> wrote:

>>>

>>>> Dear Community,

>>>>

>>>>

>>>> I am doing a reliability study, using the methods of

>>>> https://www.ncbi.nlm.nih.gov/pubmed/28505546. I have a question on

>>>> the lmer formulation and the use of the variance components.

>>>>

>>>> Background: I have 20 subjects, 2 fixed raters, 2 testing sessions,

>>>> and

>>>> 10 trials per sessions. my dependent variable is a continuous

>>>> variable (scale 1-10). Sessions are nested within each

>>>> subject-assessor combination. I desire a ICC (3) formulation of

>>>> inter-rater and inter-session reliability from the variance components.

>>>>

>>>> My lmer model is:

>>>>

>>>> lmer (dv ~ rater + (1|subj) + (1|subj:session), data = df)

>>>>

>>>> Question:

>>>>

>>>>   1.  is the model formulation right? and is my interpretation of

>>>>the  variance components for ICC below right?

>>>>   2.  inter-rater ICC = var (subj) / (var(subj) + var (residual)) #

>>>>I  read that the variation of raters will be lumped with the residual

>>>>   3.  inter-session ICC =( var (subj) + var (residual)) /( var

>>>>(subj) +  var (subj:session) + var (residual))  some simulated data:

>>>> df = expand.grid(subj = c(1:20), rater = c(1:2), session = c(1:2),

>>>>trial  = c(1:10))  df$vas = rnorm (nrow (df_sim), mean = 3, sd =

>>>>1.5)

>>>>

>>>> I appreciate the kind response.

>>

>>

>

> _______________________________________________

> R-sig-mixed-models using r-project.org<mailto:R-sig-mixed-models using r-project.org> mailing list

> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

	[[alternative HTML version deleted]]