[R] crr - computationally singular

[Ravi Varadhan]

Hi Laura,

I do think that the problem occurs during the iterations of the Newton's method for solving the estimating equations in the Fine & Gray model.  One way to overcome this would be to use the generalized inverse in the Newton algorithm. But this would yield non-unique, possibly large, coefficients, but this model would still be okay for prediction.  

My advice would be to work with fewer predictors. If you do not like this option, you should contact Bob Gray at Harvard, who is the author of this package, to get advice on how to solve your problem.

Ravi.

____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu


----- Original Message -----
From: Peter Dalgaard <P.Dalgaard at biostat.ku.dk>
Date: Monday, July 6, 2009 8:08 am
Subject: Re: [R] crr - computationally singular
To: Laura Bonnett <l.j.bonnett at googlemail.com>
Cc: r-help at r-project.org, Ravi Varadhan <rvaradhan at jhmi.edu>


> Laura Bonnett wrote:
>  > Hi Everyone,
>  > 
>  > Thank you for all your comments and suggestions.
>  > 
>  > I determined that I had a full rank model matrix by using the code:
>  >> qr(covaeb)$rank
>  > This is 17 which is equal to the number of covariates in the 
> matrix, covaeb.
>  > 
>  > I cannot invert the model matrix using 'solve' as my matrix is not
>  > square.  
>  
>  Ravi was posibly not fully awake when he suggested that...
>  The singular values (SVD(X)$d) could be more informative.
>  
>  It is possible that the function is using a less sophisticated matrix
>  inversion than those based on QR or SVD, in which case they may also
>  disagree on the rank. Otherwise, as Ravi suggested, this sort of thing
>  can also happen during (divergent) iterations with e.g. some
>  observations ending up with zero weights.
>  
>  -pd
>  
>  BTW, there's no 'crr' package that I can find. You meant 'cmprisk'?
>  
>  > In the matrices ending in a, there are 1677 rows and 15
>  > columns/covariates while in the matrices ending in b, there are 701
>  > rows and 17 columns.
>  > 
>  > Thank you,
>  > 
>  > Laura
>  > 
>  > 2009/6/26 Ravi Varadhan <RVaradhan at jhmi.edu>:
>  >> How did you determine that you have "full rank" model matrix 
> comprising 17
>  >> predictors?  Are you able to invert the model matrix using 
> `solve'?  If not,
>  >> you still have collinearity problem.
>  >>
>  >> If you are, then the problem might be in the Newton's method used 
> by `crr'
>  >> to solve the partial-likelihood optimization.  The hessian matrix 
> of the
>  >> parameters might be singular during the iterations.  If this is 
> the case,
>  >> your best bet would be to just simplify the model, i.e. use fewer
>  >> predictors.
>  >>
>  >> Ravi.
>  >>
>  >
>  
>  -- 
>     O__  ---- Peter Dalgaard             Øster Farimagsgade 5, Entr.B
>    c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
>   (*) \(*) -- University of Copenhagen   Denmark      Ph:  (+45) 35327918
>  ~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)              FAX: (+45) 35327907
>  
>  ______________________________________________
>  R-help at r-project.org mailing list
>  
>  PLEASE do read the posting guide 
>  and provide commented, minimal, self-contained, reproducible code.