[R] Memory management in R

[Mike Marchywka]

----------------------------------------
> From: dwinsemius at comcast.net
> To: lorenzo.isella at gmail.com
> Date: Fri, 8 Oct 2010 19:30:45 -0400
> CC: r-help at r-project.org
> Subject: Re: [R] Memory management in R
>
>
> On Oct 8, 2010, at 6:42 PM, Lorenzo Isella wrote:
>

> > Please find below the R snippet which requires an input file (a
> > simple text file) you can download from
> >
> > http://dl.dropbox.com/u/5685598/time_series25_.dat
> >
> > What puzzles me is that the list is not really long (less than 2000
> > entries) and I have not experienced the same problem even with
> > longer lists.
>
> But maybe your loop terminated in them eaarlier/ Someplace between
> 11*225 and 11*240 the grepping machine gives up:
>
> > eprs <- paste(rep("aaaaaaaaaa", 225), collapse="#")
> > grepl(eprs, eprs)
> [1] TRUE
>
> > eprs <- paste(rep("aaaaaaaaaa", 240), collapse="#")
> > grepl(eprs, eprs)
> Error in grepl(eprs, eprs) :
> invalid regular expression
> 'aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaaaaaaa#aaaaa
> In addition: Warning message:
> In grepl(eprs, eprs) : regcomp error: 'Out of memory'
>
> The complexity of the problem may depend on the distribution of
> values. You have a very skewed distribution with the vast majority
> being in the same value as appeared in your error message :
>

>
> HTH (although I think it means you need to construct a different
> implementation strategy);

You really need to look at the question posed by your regex and consider 
the complexity of what you are asking and what likely implementations
would do with your regex. Something like this probably needs to be implemented
in dedicated code to handle the more general case or you need to determine
if input data is pathological given your regex. Being able to write something
concisely doesn't mean the execution of that something is simple. Even if
it does manage to return a result, it likely will get very slow. In the
past I have had to write my own simple regex compilers to handle a limited
class of expressions to make the speed reasonable. In this case, depending
on your objectives, dedicated code may even be helpful to you in understanding
the algorithm. 

>
> David.
>
>
> > Many thanks
> >
> > Lorenzo
> >