[R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

[Ben Bolker]

More generally, the best way to fit this kind of model is to use an
*ordinal* model, which assumes the responses are in increasing
sequence but does not assume the distance between levels (e.g. 1 vs 2,
2 vs 3 ...) is uniform.  However, I'm not sure how one would go about
computing an ICC from ordinal data ... (the 'ordinal' package is the
place to look for the model-fitting procedures). Googling it finds
some stuff, but it seems that it doesn't necessarily apply to complex
designs ...

https://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/


On Thu, Jun 14, 2018 at 6:58 PM, Doran, Harold <HDoran using air.org> wrote:
> That’s a helpful clarification, Rolf. However, with gaussian normal errors
> in the linear model, we can’t *really* assume they would asymptote at 1 or
> 10. My suspicion is that these are likert-style ordered counts of some
> form, although the OP should clarify. In which case, the 1 or 10 are
> limits with censoring, as true values for some measured trait could exist
> outside those boundaries (and I suspect the model is forming predicted
> values outside of 1 or 10).
>
>
>
> On 6/14/18, 6:33 PM, "Rolf Turner" <r.turner using auckland.ac.nz> wrote:
>
>>
>>On 15/06/18 05:35, Doran, Harold wrote:
>>
>>> Well no, you¹re specification is not right because your variable is not
>>> continuous as you note. Continuous means it is a real number between
>>> -Inf/Inf and you have boundaries between 1 and 10. So, you should not be
>>> using a linear model assuming the outcome is continuous.
>>
>>I think that the foregoing is a bit misleading.  For a variable to be
>>continuous it is not necessary for it to have a range from -infinity to
>>infinity.
>>
>>The OP says that dv  "is a continuous variable (scale 1-10)".  It is not
>>clear to me what this means.  The "obvious"/usual meaning or
>>interpretation would be that dv can take (only) the (positive integer)
>>values 1, 2, ..., 10.  If this is so, then a continuous model is not
>>appropriate.  (It should be noted however that people in the social
>>sciences do this sort of thing --- i.e. treat discrete variables as
>>continuous --- all the time.)
>>
>>It is *possible* that dv can take values in the real interval [1,10], in
>>which case it *is* continuous, and a "continuous model" is indeed
>>appropriate.
>>
>>The OP should clarify what the situation actually is.
>>
>>cheers,
>>
>>Rolf Turner
>>
>>--
>>Technical Editor ANZJS
>>Department of Statistics
>>University of Auckland
>>Phone: +64-9-373-7599 ext. 88276
>>
>>> On 6/14/18, 11:16 AM, "Bernard Liew" <B.Liew using bham.ac.uk> wrote:
>>>
>>>> Dear Community,
>>>>
>>>>
>>>> I am doing a reliability study, using the methods of
>>>> https://www.ncbi.nlm.nih.gov/pubmed/28505546. I have a question on the
>>>> lmer formulation and the use of the variance components.
>>>>
>>>> Background: I have 20 subjects, 2 fixed raters, 2 testing sessions, and
>>>> 10 trials per sessions. my dependent variable is a continuous variable
>>>> (scale 1-10). Sessions are nested within each subject-assessor
>>>> combination. I desire a ICC (3) formulation of inter-rater and
>>>> inter-session reliability from the variance components.
>>>>
>>>> My lmer model is:
>>>>
>>>> lmer (dv ~ rater + (1|subj) + (1|subj:session), data = df)
>>>>
>>>> Question:
>>>>
>>>>   1.  is the model formulation right? and is my interpretation of the
>>>> variance components for ICC below right?
>>>>   2.  inter-rater ICC = var (subj) / (var(subj) + var (residual)) # I
>>>> read that the variation of raters will be lumped with the residual
>>>>   3.  inter-session ICC =( var (subj) + var (residual)) /( var (subj) +
>>>> var (subj:session) + var (residual))
>>>> some simulated data:
>>>> df = expand.grid(subj = c(1:20), rater = c(1:2), session = c(1:2),
>>>>trial
>>>> = c(1:10))
>>>> df$vas = rnorm (nrow (df_sim), mean = 3, sd = 1.5)
>>>>
>>>> I appreciate the kind response.
>>
>>
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models