[Bioc-devel] Increasing MAX_BUFLENGTH in S4Vectors src/AEbufs.c

[Hervé Pagès]

Hi Pete,

On 07/19/2016 06:59 PM, Peter Hickey wrote:
> I hit an error when calling reduce() on a very big GRanges object
> (length = 1170402558). The error was:
>
> Error in .Call2("CompressedIRangesList_reduce", x, drop.empty.ranges,  :
>    _get_new_buflength(): MAX_BUFLENGTH reached
>
> I found MAX_BUFLENGTH is defined in S4Vectors in the file src/AEbufs.c as:
>
> #define MAX_BUFLENGTH_INC (32 * 1024 * 1024)
> #define MAX_BUFLENGTH (32 * MAX_BUFLENGTH_INC)
>
> So I experimentally increased the limit in a local copy. I first set
> it to twice the current value (which errored on R CMD check, I think
> because that makes MAX_BUFLENGTH > the maximum allowable integer), but
> by setting it to 1.5 times its current limit I got (A) an apparently
> working copy of S4Vectors and (B) the original code now ran without
> error.
>
> So, is it safe to increase MAX_BUFLENGTH or am I missing some important details?

Yes it's safe, as long as it remains <= .Machine$integer.max, as you
found out. I just increased its value to 63 * 2^25 in S4Vectors 0.11.10.
When we start supporting long Vector derivatives, we'll be able to
increase it way beyond .Machine$integer.max

Cheers,
H.

>
> Thanks,
> Pete
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319