[R] R: "in-place" appending to a matrix.

[Gabor Grothendieck]

The real problem is that running a loop to build up an object bit by bit
is not the preferred way to write an R program.  The preferred way
is to use lapply or other apply command in which case the object
gets built for you.

On Thu, Oct 16, 2008 at 10:50 AM, culpritNr1 <ig2ar-saf1 at yahoo.co.uk> wrote:
>
> Hello Gabor,
>
> First of all, thanks for your reply.
>
> Indeed, your strategy solves the problem of speed=). My question was,
> however, more of an R design question.
>
> Your suggestion of pre-allocating a big and continuous portion of memory is
> probably the first manual work around to try. And it will work.
>
> Now, modern high level languages like the continually improving R, Python
> and also Matlab and Perl come with built-in goodies that run at C/C++ speed.
> Those built-in functions are high level versions of functionalities that we
> would code "manually" in low level languages.
>
> Being relatively new to R, my expectations are that in a high level language
> like this, I do not need to manually program a simple operation such as
> in-place appending. Manipulation of rectangular data should be extremely
> efficient. (I use it to make computation of DNA properties in organisms with
> huge genomes like human and mouse.)
>
> So, do you know whether or not such functions exists or is planned to be
> incorporated in the near future? An optional module perhaps?
>
> Again, thank you,
>
> culpritNr1
>
>
>
>
>
> Gabor Grothendieck wrote:
>>
>> Create an empty matrix first and then fill it in.   That
>> will avoid the overhead in repeatedly expanding it.  If
>> you don't know how many rows then make it 1000 rows
>> and remove the unused ones once finished.
>>
>> On Wed, Oct 15, 2008 at 4:05 PM, culpritNr1 <ig2ar-saf1 at yahoo.co.uk>
>> wrote:
>>>
>>> Hello fellow R sufferers,
>>>
>>> Is there a way to perform an appending operation in place?
>>>
>>> Currently, the way my pseudo-code goes is like this
>>>
>>> for (i in 1:1000) {
>>>    if (some condition) {
>>>        newRow <- myFunction(myArguments)
>>>        X <- rbind(X, newRow)     #  <- this is the bottleneck!!
>>>    }
>>> }
>>>
>>> As you can see, it works but as the matrix X gets the size of a few
>>> million
>>> rows, the code runs very slow.
>>>
>>> I am looking for something like the natively "in place" appending python
>>> function called "append()" or the perl function "push". "In-place"
>>> operations would allow me to do (in pseudocode)
>>>
>>> for (i in 1:1000) {
>>>    if (some condition) {
>>>        newRow <- myFunction(myArguments)
>>>        append(X, newRow)
>>>    }
>>> }
>>>
>>> You see? I do not have to call and re-assign the giant X matrix every
>>> loop
>>> cycle.
>>>
>>> Any help?
>>>
>>> Thank you,
>>>
>>> Your culpritNr1
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/R%3A-%22in-place%22-appending-to-a-matrix.-tp20001258p20001258.html
>>> Sent from the R help mailing list archive at Nabble.com.
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/R%3A-%22in-place%22-appending-to-a-matrix.-tp20001258p20014954.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>