[R] Slow reading multiple tick data files into list of dataframes

[Mike Marchywka]

----------------------------------------
> Date: Mon, 11 Oct 2010 14:39:54 -0700
> From: aquanyc at gmail.com
> To: r-help at r-project.org
> Subject: [R] Slow reading multiple tick data files into list of dataframes
[...]
> Is there a better/quicker or more R way of doing this ?

While there may be an obvious R-related answer, usually it helps if you 
can determine where the bottleneck is in terms of 
resources on your platform- often on older machines you
run out of real memory and then all the time is spent reading
the file onto VM back on disk. Can you tell if you are CPU or
memory limited by using task manager? 

It could in fact be that the best solution involves not trying
to hold your entire data set in memory at once, hard to know without
knowing your platform etc. In the
past, I've found that actually sorting data, a slow process
itself, can speed things up a lot due to less thrashing
of memory hierarchy during the later analysis. I doubt 
if that helps your immediate problem but it does point
to one possible non-obvious "optimization" depending
on what is slowing you down.


>
> Thanks,
> Chris
>
> --
> View this message in context: http://r.789695.n4.nabble.com/Slow-reading-multiple-tick-data-files-into-list-of-dataframes-tp2990723p2990723.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.