[R] Speeding up prediction of survival estimates when using `survifit'

[Ravi Varadhan]

Hi,

I fit a Cox PH model to estimate the cause-specific hazards (in a competing risks setting).  Then , I compute the survival estimates for all the individuals in my data set using the `survfit' function.  I am currently playing with a data set that has about 6000 observations and 12 covariates.  I am finding that the survfit function is very slow.  

Here is a simple simulation example (modified from Frank Harrell's example for `cph') that illustrates the problem:

#n <- 500
set.seed(4321) 

age <- 50 + 12*rnorm(n) 

sex <- factor(sample(c('Male','Female'), n, rep=TRUE, prob=c(.6, .4))) 

cens <- 5 * runif(n) 

h <- 0.02 * exp(0.04 * (age-50) + 0.8 * (sex=='Female')) 

dt <- -log(runif(n))/h 

e <- ifelse(dt <= cens, 1, 0) 

dt <- pmin(dt, cens) 

Srv <- Surv(dt, e)

 f <- coxph(Srv ~ age + sex, x=TRUE, y=TRUE) 

system.time(ans <- survfit(f, type="aalen", se.fit=FALSE, newdata=f$x))


When I run the above code with sample sizes, n, taking on values of 500, 1000, 2000, and 4000, the time it takes for survfit to run are as follows:

# n <- 500
> system.time(ans <- survfit(f, type="aalen", se.fit=FALSE, newdata=f$x))
   user  system elapsed 
   0.16    0.00    0.15 


# n <- 1000
> system.time(ans <- survfit(f, type="aalen", se.fit=FALSE, newdata=f$x))
   user  system elapsed 
   1.45    0.00    1.48 


# n <- 2000
> system.time(ans <- survfit(f, type="aalen", se.fit=FALSE, newdata=f$x))
   user  system elapsed 
  10.19    0.00   10.25 


# n <- 4000
> system.time(ans <- survfit(f, type="aalen", se.fit=FALSE, newdata=f$x))
   user  system elapsed 
  72.87    0.05   74.87 


I eventually want to use `survfit' on a data set with roughly 50K observations, which I am afraid is going to be painfully slow.  I would much appreciate hints/suggestions on how to make `survfit' faster or any other faster alternatives.  

Thanks.

Best,
Ravi.
____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu