[R] Effeciently sum 3d table

Mon Apr 16 22:38:23 CEST 2012

I generally prefer the list approach too.  I only mentioned that you didn't
need to have a list of inputs before starting the summation because
you said
   > There's a lot
  > of extraneous memory wastage holding all of the tables in a list as well as
  > the return 'values'.
I guess I misinterpreted that sentence.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

From: David A Vavra [mailto:davavra at verizon.net] 
Sent: Monday, April 16, 2012 1:20 PM
To: William Dunlap
Cc: r-help at r-project.org
Subject: RE: [R] Effeciently sum 3d table

Thanks Bill,

For reasons that aren't important here, I must start from a list. Computing the sum while generating the tables may be a solution but it means doing something in one piece of code that is unrelated to the surrounding code. Bad practice where I'm from. If it's needed it's needed but if I can avoid doing so, I will. 

I haven't done any timing but because of the extra operations of get and assign, the non-loop implementation will likely suffer. It seems you have shown this to be true.

DAV

-----Original Message-----
From: William Dunlap [mailto:wdunlap at tibco.com] 
Sent: Monday, April 16, 2012 3:26 PM
To: David A Vavra; 'Bert Gunter'
Cc: r-help at r-project.org
Subject: RE: [R] Effeciently sum 3d table

> Example in partial code:
> 
> Env <- CreatEnv() # my own function
> Assign('final',T1-T1,envir=env)
> L<-listOfTables
> 
> lapply(L,function(t) {
>     final <- get('final',envir=env) + t
>     assign('final',final,envir=env)
>     NULL
> })

First, finish writing that code so it runs and you can make sure its
output is ok:

L <- lapply(1:50000, function(i) array(i:(i+3), c(2,2))) # list of 50,000 2x2 matrices
env <- new.env()
assign('final', L[[1]] - L[[1]], envir=env)
junk <- lapply(L, function(t) {
     final <- get('final', envir=env) + t
     assign('final', final, envir=env)
     NULL
})
get('final', envir=env)
#            [,1]       [,2]
# [1,] 1250025000 1250125000
# [2,] 1250075000 1250175000
> sum( (2:50001) ) # should be final[2,1]
# [1] 1250075000

You asked for something less "clunky".
You are fighting the system by using get() and assign(), just use
ordinary expression syntax to get and set variables:
final <- L[[1]]
for(i in seq_along(L)[-1]) final <- final + L[[i]]
final
#           [,1]       [,2]
# [1,] 1250025000 1250125000
# [2,] 1250075000 1250175000

The former took 0.22 seconds on my machine, the latter 0.06.

You don't have to compute the whole list of matrices before
doing the sum, just add to the current sum when you have
computed one matrix and then forget about it.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf
> Of David A Vavra
> Sent: Monday, April 16, 2012 11:35 AM
> To: 'Bert Gunter'
> Cc: r-help at r-project.org
> Subject: Re: [R] Effeciently sum 3d table
> 
> Thanks Gunter,
> 
> I mean what I think is the normal definition of 'sum' as in:
>    T1 + T2 + T3 + ...
> It never occurred to me that there would be a question.
> 
> I have gotten the impression that a for loop is very inefficient. Whenever I
> change them to lapply calls there is a noticeable improvement in run time
> for whatever reason. The problem with lapply here is that I effectively need
> a global table to hold the final sum. lapply also  wants to return a value.
> 
> You may be correct that in the long run, the loop is the best. There's a lot
> of extraneous memory wastage holding all of the tables in a list as well as
> the return 'values'.
> 
> As an alternate and given a pre-existing list of tables, I was thinking of
> creating a temporary environment to hold the final result so it could be
> passed globally to each lapply execution level but that seems clunky and
> wasteful as well.
> 
> Example in partial code:
> 
> Env <- CreatEnv() # my own function
> Assign('final',T1-T1,envir=env)
> L<-listOfTables
> 
> lapply(L,function(t) {
>     final <- get('final',envir=env) + t
>     assign('final',final,envir=env)
>     NULL
> })
> 
> But I was hoping for a more elegant and hopefully more efficient solution.
> Greg's suggestion for using reduce seems in order but as yet I'm unfamiliar
> with the function.
> 
> DAV
> 
> 
> 
> -----Original Message-----
> From: Bert Gunter [mailto:gunter.berton at gene.com]
> Sent: Monday, April 16, 2012 12:42 PM
> To: Greg Snow
> Cc: David A Vavra; r-help at r-project.org
> Subject: Re: [R] Effeciently sum 3d table
> 
> Define "sum" . Do you mean you want to get a single sum for each
> array? -- get marginal sums for each array? -- get a single array in
> which each value is the sum of all the individual values at the
> position?
> 
> Due thought and consideration for those trying to help by formulating
> your query carefully and concisely vastly increases the chance of
> getting a useful answer. See the posting guide -- this is a skill that
> needs to be learned and the guide is quite helpful. And I must
> acknowledge that it is a skill that I also have not yet mastered.
> 
> Concerning your query, I would only note that the two responses from
> Greg and Petr that you received are unlikely to be significantly
> faster than just using loops, since both are still essentially looping
> at the interpreted level. Whether either give you what you want, I do
> not know.
> 
> -- Bert
> 
> On Mon, Apr 16, 2012 at 8:53 AM, Greg Snow <538280 at gmail.com> wrote:
> > Look at the Reduce function.
> >
> > On Mon, Apr 16, 2012 at 8:28 AM, David A Vavra <davavra at verizon.net>
> wrote:
> >> I have a large number of 3d tables that I wish to sum
> >> Is there an efficient way to do this? Or perhaps a function I can call?
> >>
> >> I tried using do.call("sum",listoftables) but that returns a single
> value.
> >>
> >> So far, it seems only a loop will do the job.
> >>
> >>
> >> TIA,
> >> DAV
> 
> 
> --
> 
> Bert Gunter
> Genentech Nonclinical Biostatistics
> 
> Internal Contact Info:
> Phone: 467-7374
> Website:
> http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biost
> atistics/pdb-ncb-home.htm
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.