[R-sig-ME] [FORGED] Re: Using variance components of lmer for ICC computation in reliability study

[Pierre de Villemereuil]

Hi,

> However, I'm not sure how one would go about computing an ICC from ordinal data

I've never used the package "ordinal", but if it's anything like the "ordinal" family of MCMCglmm, then computing an ICC on the liability scale would be fairly easy, as one would just proceed as always and simply add the so-called "link variance" corresponding to the chosen link function (1 for probit, (pi^2)/3 for logit). E.g. for a given variance component Vcomp and a probit link:
ICC = Vcomp / (sum(variance components of the model) + 1)

However, computing an ICC on the data scale would be much more difficult as it is actually multivariate...

I think in the case when such scores were used, having the estimate on the liability scale make sense though, as the binning is more due to our inability of finely measuring this scale rather than an actual property of the system.

Cheers,
Pierre.

Le vendredi 15 juin 2018, 03:27:54 CEST Ben Bolker a écrit :
> More generally, the best way to fit this kind of model is to use an
> *ordinal* model, which assumes the responses are in increasing
> sequence but does not assume the distance between levels (e.g. 1 vs 2,
> 2 vs 3 ...) is uniform.  However, I'm not sure how one would go about
> computing an ICC from ordinal data ... (the 'ordinal' package is the
> place to look for the model-fitting procedures). Googling it finds
> some stuff, but it seems that it doesn't necessarily apply to complex
> designs ...
> 
> https://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data
> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/
> 
> 
> On Thu, Jun 14, 2018 at 6:58 PM, Doran, Harold <HDoran using air.org> wrote:
> > That’s a helpful clarification, Rolf. However, with gaussian normal errors
> > in the linear model, we can’t *really* assume they would asymptote at 1 or
> > 10. My suspicion is that these are likert-style ordered counts of some
> > form, although the OP should clarify. In which case, the 1 or 10 are
> > limits with censoring, as true values for some measured trait could exist
> > outside those boundaries (and I suspect the model is forming predicted
> > values outside of 1 or 10).
> >
> >
> >
> > On 6/14/18, 6:33 PM, "Rolf Turner" <r.turner using auckland.ac.nz> wrote:
> >
> >>
> >>On 15/06/18 05:35, Doran, Harold wrote:
> >>
> >>> Well no, you¹re specification is not right because your variable is not
> >>> continuous as you note. Continuous means it is a real number between
> >>> -Inf/Inf and you have boundaries between 1 and 10. So, you should not be
> >>> using a linear model assuming the outcome is continuous.
> >>
> >>I think that the foregoing is a bit misleading.  For a variable to be
> >>continuous it is not necessary for it to have a range from -infinity to
> >>infinity.
> >>
> >>The OP says that dv  "is a continuous variable (scale 1-10)".  It is not
> >>clear to me what this means.  The "obvious"/usual meaning or
> >>interpretation would be that dv can take (only) the (positive integer)
> >>values 1, 2, ..., 10.  If this is so, then a continuous model is not
> >>appropriate.  (It should be noted however that people in the social
> >>sciences do this sort of thing --- i.e. treat discrete variables as
> >>continuous --- all the time.)
> >>
> >>It is *possible* that dv can take values in the real interval [1,10], in
> >>which case it *is* continuous, and a "continuous model" is indeed
> >>appropriate.
> >>
> >>The OP should clarify what the situation actually is.
> >>
> >>cheers,
> >>
> >>Rolf Turner
> >>
> >>--
> >>Technical Editor ANZJS
> >>Department of Statistics
> >>University of Auckland
> >>Phone: +64-9-373-7599 ext. 88276
> >>
> >>> On 6/14/18, 11:16 AM, "Bernard Liew" <B.Liew using bham.ac.uk> wrote:
> >>>
> >>>> Dear Community,
> >>>>
> >>>>
> >>>> I am doing a reliability study, using the methods of
> >>>> https://www.ncbi.nlm.nih.gov/pubmed/28505546. I have a question on the
> >>>> lmer formulation and the use of the variance components.
> >>>>
> >>>> Background: I have 20 subjects, 2 fixed raters, 2 testing sessions, and
> >>>> 10 trials per sessions. my dependent variable is a continuous variable
> >>>> (scale 1-10). Sessions are nested within each subject-assessor
> >>>> combination. I desire a ICC (3) formulation of inter-rater and
> >>>> inter-session reliability from the variance components.
> >>>>
> >>>> My lmer model is:
> >>>>
> >>>> lmer (dv ~ rater + (1|subj) + (1|subj:session), data = df)
> >>>>
> >>>> Question:
> >>>>
> >>>>   1.  is the model formulation right? and is my interpretation of the
> >>>> variance components for ICC below right?
> >>>>   2.  inter-rater ICC = var (subj) / (var(subj) + var (residual)) # I
> >>>> read that the variation of raters will be lumped with the residual
> >>>>   3.  inter-session ICC =( var (subj) + var (residual)) /( var (subj) +
> >>>> var (subj:session) + var (residual))
> >>>> some simulated data:
> >>>> df = expand.grid(subj = c(1:20), rater = c(1:2), session = c(1:2),
> >>>>trial
> >>>> = c(1:10))
> >>>> df$vas = rnorm (nrow (df_sim), mean = 3, sd = 1.5)
> >>>>
> >>>> I appreciate the kind response.
> >>
> >>
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> 
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models