[Rd] R's IO speed

[Prof Brian Ripley]

Use memory.size(max=TRUE) in Windows, or some other profiling memory 
manager.

On Mon, 3 Jan 2005, Vadim Ogranovich wrote:

> A technical question here: how does one measure the memory overhead
> mentioned below? I have a set of functions of my own and would like to
> profile them.
>
> Thanks,
> Vadim
>
>> -----Original Message-----
>> From: r-devel-bounces at stat.math.ethz.ch
>> [mailto:r-devel-bounces at stat.math.ethz.ch] On Behalf Of Prof
>> Brian Ripley
>> Sent: Sunday, December 26, 2004 2:04 AM
>> To: R-devel at r-project.org
>> Subject: [Rd] R's IO speed
>>
>> R-devel now has some improved versions of read.table and write.table.
>>
>> For a million-row data frame containing one number, one
>> factor with few levels and one logical column, a 56Mb object.
>>
>> generating it takes 4.5 secs.
>>
>> calling summary() on it takes 2.2 secs.
>>
>> writing it takes 8 secs and an additional 10Mb.
>>
>> saving it in .rda format takes 4 secs.
>>
>> reading it naively takes 28 secs and an additional 240Mb
>>
>> reading it carefully (using nrows, colClasses and
>> comment.char) takes 16 secs and an additional 150Mb (56Mb of
>> which is for the object read in).
>> (The overhead of read.table over scan was about 2 secs,
>> mainly in the conversion back to a factor.)
>>
>> loading from .rda format takes 3.4 secs.
>>
>> [R 2.0.1 read in 23 secs using an additional 210Mb, and wrote
>> in 50 secs using an additional 450Mb.]
>>
>>
>> Will Frank Harrell or someone else please explain to me a
>> real application
>> in which this is not fast enough?
>>
>> --
>> Brian D. Ripley,                  ripley at stats.ox.ac.uk
>> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>> University of Oxford,             Tel:  +44 1865 272861 (self)
>> 1 South Parks Road,                     +44 1865 272866 (PA)
>> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>>
>> ______________________________________________
>> R-devel at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595