[Bioc-devel] Tip of the day: unlist(..., use.names=FALSE) often saves lots of memory

[Henrik Bengtsson]

Hi Martin,

thanks for your important comments.  I knew it was coming - I am aware
of Seth's addition of string suffix trees, which indeed saves lots of
memory and some overhead.  However, I have some comments below.

On Sat, Jul 5, 2008 at 6:48 PM, Martin Morgan <mtmorgan at fhcrc.org> wrote:
> Hi Henrik --
>
> "Henrik Bengtsson" <hb at stat.berkeley.edu> writes:
>
>> Hi,
>>
>> I just wanna share an seldom used feature of unlist():
>>
>>   Using argument 'use.names=FALSE' when calling unlist() often saves
>> lots of memory.
>
> Actually, thanks to some cleverness introduced largely by Seth, the
> savings might be less than you think...
>
>> The names vector of the list will be expanded to each element and can
>> often consume much more memory than the actually data.  So, unless you
>> really need the 'names' attributes, please consider using unlist(...,
>> use.names=FALSE) in your package(s).  It is also faster.
>>
>> A common example using an AffyBatch object:
>>
>>> affyBatch
>> AffyBatch object
>> size of arrays=1164x1164 features (7 kb)
>> cdf=HG-U133_Plus_2 (54675 affyids)
>> number of samples=1
>> number of genes=54675
>> annotation=hgu133plus2
>> notes=
>
> affyBatch already has a copy of each probe name. R has made an
> internal hash of all unique character strings (this will always be
> true when use.names=FALSE might be useful -- the names will already
> exist), so here...

I just used the AffyBatch class as an example, so I don't really want
to dig into details about that class.  Here is a more general example:

> x <- list(a=1:4, b=6:7)
> unlist(x)
a1 a2 a3 a4 b1 b2
 1  2  3  4  6  7

The names attribute of 'x' is two strings, but when unlist():ed the
names are expanded, used a prefixes and enumerated.  A suffix tree
will of course save some memory here, but it will still require new
strings to be created.

About AffyBatch, does it actually store these "extended" names:

> head(names(unlist(pmIndex)), 20)
 [1] "1007_s_at1"  "1007_s_at2"  "1007_s_at3"  "1007_s_at4"  "1007_s_at5"
 [6] "1007_s_at6"  "1007_s_at7"  "1007_s_at8"  "1007_s_at9"  "1007_s_at10"
[11] "1007_s_at11" "1007_s_at12" "1007_s_at13" "1007_s_at14" "1007_s_at15"
[16] "1007_s_at16" "1053_at1"    "1053_at2"    "1053_at3"    "1053_at4"

or just the probeset names:

> head(names(pmIndex))
[1] "1007_s_at" "1053_at"   "117_at"    "121_at"    "1255_g_at" "1294_at"

>
>>> pmIndex <- indexProbes(affyBatch[,1], "pm")
>
> ...you make copies of the references to the names, not of the names
> themselves. And ...
>
>>> object.size(pmIndex)
>> [1] 6572776
>>
>>> cells <- unlist(pmIndex)
>>> object.size(cells)
>> [1] 29018704
>
> ... here R is counting the size of the object and the size of the
> names in the cache, even though the memory footprint of the cached
> names are in some sense amortized over affyBatch, pmIndex, and
> cells. A different estimate of the cost would be to compare
>
> cells3 <- cells2
> names(cells3) <- ""
> object.size(cells3) / object.size(cells2)
>
> This reflects the cost of the underlying pointer to the character
> string, with the character string itself costing almost nothing.
>
> On the 64 bit machine I'm working on now,
>
>> object.size(character(1024^2)) / object.size(integer(1024^2))
> [1] 2.000002
>
> so an element of a character vector takes up about twice as much space
> as an element of an integer vector. I'd expect the ratio of the sizes
> of cells3 / cells2 to be about (1 + 2) / 1 = 3, so adding names
> triples the object size. On my 32 bit laptop or if cells were numeric,
> the size only doubles.
>
>>> cells2 <- unlist(pmIndex, use.names=FALSE)
>>> object.size(cells2)
>> [1] 2417056
>>
>> # The names consumes 92% of the memory
>>> object.size(cells2)/object.size(cells)
>> [1] 0.08329304
>
>> It is much cheaper to pass around 'cells2' compared with 'cells'.
>
> ... R's approximate copy on change semantics makes it quite difficult
> to know whether this is really true or not -- a variable passed to a
> function and used in a read-only fashion is unlikely to be copied, so
> 'passing around' is really light-weight (this changes with S4, but
> that is an implementation issue that might some day be fully
> resolved).
>
> On the other hand, dropping names makes, in my experience, subsetting
> and other data coordination errors significantly more likely, and I've
> usually regretted trying to be efficient in this way -- it's working
> against the software, instead of with it.
>
> Creation of new names, or checking whether new names need to be
> created, can be quite time-consuming, for instance when data frame row
> names are created (during, e.g., write.table), or numeric values
> converted to characters (e.g., comparing integer and character
> values). In your example above, I found that using unlist(pmIndex,
> use.names=FALSE) actually lead to a 10x speedup, but since this was
> from 0.1 to 0.01 seconds. I don't know that this is worth it for
> interactive calculation on data the size of 'standard' expression
> arrays. Perhaps in a heavily used function where I know that the
> nameless entity will not come back to get me, or when data gets truly
> big; definitely there are situations where use.names=FALSE seems to be
> a big help.

In our experience developing/using aroma.affymetrix, we (not the royal
one this time) found that unlist(..., use.names=FALSE) saves a lot of
memory and seems to speed things up, e.g. when working with nested CDF
list structures from affxparser.  Also, we found by looking at the
internal code that we very rarely used the names attributes so we
found that discarding them ASAP to be a better strategy.  All our
indexing is done by integer indices and never by names; that was an
early design decision.  We have other ways to validate the correctness
of our algorithms.  When I look at BioC code (and elsewhere), it is
not-uncommon that the names attributes are not used for anything good,
and sometimes they are discarded *at the very end* whereas they
equally well could have been discarded from the beginning.

Cheers

Henrik

>
> Martin
>
>>
>> /Henrik
>>
>> _______________________________________________
>> Bioc-devel at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
> --
> Martin Morgan
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M2 B169
> Phone: (206) 667-2793
>