[R] read big text file into R
Charles C. Berry
cberry at tajo.ucsd.edu
Thu Aug 23 21:46:36 CEST 2007
On Thu, 23 Aug 2007, Yupu Liang wrote:
> Dear Rs:
>
> Hi, I am trying to read a big text file (nrows=243440, ncols=144). It
> seems the computational time of all the read methods
> (scan,readtable,read.delim) is not linear to the number of rows I
> want to read in: things became really slow once I tried to read in
> 100000 lines compare to 10000 lines).
What did 'top' or gc() tell you about memory use, paging, and swap
behavior?
Likely you are starting to swap when you get to nrows=100000.
Even if your data were all numeric, over 100MB would be needed just to
store them. And reading them in requires more memory still.
>
> If I am reading the profiling result right, I guess scan wouldn't
> help either.
>
> My questions are :
> 1) Is this a memory issue?
> 2) How to get around this?: I can't just sit around for 15 mins.
> Would write a c function help?
>
> Thanks!
>
> Here is the profiling I did:
>
> > Rprof()
> > dd = read.delim(file,skip=9,sep="\t",as.is= T,nrows=10000)
> > Rprof(NULL)
> > summaryRprof()
> $by.self
> self.time self.pct total.time total.pct
> "scan" 3.56 85.2 3.56 85.2
> "type.convert" 0.48 11.5 0.48 11.5
> "read.table" 0.08 1.9 4.18 100.0
> "make.names" 0.02 0.5 0.02 0.5
> "options" 0.02 0.5 0.02 0.5
> "readLines" 0.02 0.5 0.02 0.5
> "read.delim" 0.00 0.0 4.18 100.0
> "file" 0.00 0.0 0.02 0.5
> "getOption" 0.00 0.0 0.02 0.5
>
> $by.total
> total.time total.pct self.time self.pct
> "read.table" 4.18 100.0 0.08 1.9
> "read.delim" 4.18 100.0 0.00 0.0
> "scan" 3.56 85.2 3.56 85.2
> "type.convert" 0.48 11.5 0.48 11.5
> "make.names" 0.02 0.5 0.02 0.5
> "options" 0.02 0.5 0.02 0.5
> "readLines" 0.02 0.5 0.02 0.5
> "file" 0.02 0.5 0.00 0.0
> "getOption" 0.02 0.5 0.00 0.0
>
> $sampling.time
> [1] 4.18
>
> > ?Rprof()
> > Rprof()
> > dd = read.delim(file,skip=9,sep="\t",as.is= T,nrows=100000)
> > Rprof(NULL)
> > summaryRprof()
> $by.self
> self.time self.pct total.time total.pct
> "scan" 143.12 92.7 143.12 92.7
> "type.convert" 9.52 6.2 9.52 6.2
> "read.table" 1.60 1.0 154.28 99.9
> "paste" 0.02 0.0 0.08 0.1
> "textConnection" 0.02 0.0 0.04 0.0
> ".deparseOpts" 0.02 0.0 0.02 0.0
> "file" 0.02 0.0 0.02 0.0
> "make.names" 0.02 0.0 0.02 0.0
> "print.default" 0.02 0.0 0.02 0.0
> "read.delim" 0.00 0.0 154.28 99.9
> "doTryCatch" 0.00 0.0 0.08 0.1
> "gsub" 0.00 0.0 0.08 0.1
> "try" 0.00 0.0 0.08 0.1
> "tryCatch" 0.00 0.0 0.08 0.1
> "tryCatchList" 0.00 0.0 0.08 0.1
> "tryCatchOne" 0.00 0.0 0.08 0.1
> "capture.output" 0.00 0.0 0.06 0.0
> "deparse" 0.00 0.0 0.02 0.0
> "eval.with.vis" 0.00 0.0 0.02 0.0
> "evalVis" 0.00 0.0 0.02 0.0
> "print" 0.00 0.0 0.02 0.0
>
> $by.total
> total.time total.pct self.time self.pct
> "read.table" 154.28 99.9 1.60 1.0
> "read.delim" 154.28 99.9 0.00 0.0
> "scan" 143.12 92.7 143.12 92.7
> "type.convert" 9.52 6.2 9.52 6.2
> "paste" 0.08 0.1 0.02 0.0
> "doTryCatch" 0.08 0.1 0.00 0.0
> "gsub" 0.08 0.1 0.00 0.0
> "try" 0.08 0.1 0.00 0.0
> "tryCatch" 0.08 0.1 0.00 0.0
> "tryCatchList" 0.08 0.1 0.00 0.0
> "tryCatchOne" 0.08 0.1 0.00 0.0
> "capture.output" 0.06 0.0 0.00 0.0
> "textConnection" 0.04 0.0 0.02 0.0
> ".deparseOpts" 0.02 0.0 0.02 0.0
> "file" 0.02 0.0 0.02 0.0
> "make.names" 0.02 0.0 0.02 0.0
> "print.default" 0.02 0.0 0.02 0.0
> "deparse" 0.02 0.0 0.00 0.0
> "eval.with.vis" 0.02 0.0 0.00 0.0
> "evalVis" 0.02 0.0 0.00 0.0
> "print" 0.02 0.0 0.00 0.0
>
> $sampling.time
> [1] 154.36
>
> I am using R 2.5.1 for mac on a Dual 2 GHz PowerPC G5 with 1GB memory.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
Charles C. Berry (858) 534-2098
Dept of Family/Preventive Medicine
E mailto:cberry at tajo.ucsd.edu UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/ La Jolla, San Diego 92093-0901
More information about the R-help
mailing list