[R-pkg-devel] Replacement for SETLENGTH

[Ivan Krylov]

On Wed, 15 Jan 2025 15:30:36 +0000
"Merlise Clyde, Ph.D." <clyde using duke.edu> wrote:

> For memory limited machines, the alloc/copy was a problem for memory
> usage - and if I recall was one of the reasons in 2008 I switched to
> SETLENGTH, which doesn't seem to do an allocation ???  If there is
> going to be an absolute ban on SETLENGTH  in packages I'll probably
> need to address memory management differently for those cases.

If you need to adjust the xlength() of your vectors without causing
reallocations, I'm afraid the only API-compliant way to do that for now
is ALTREP [1]. It's a lot of typing because the code will need to
register an ALTREP class for every vector type that needs to be
shrinkable. Moreover, since some of those vectors are of type VECSXP
and "altlist" classes only appeared in R-4.3.0, you will probably
prefer to leave the old implementation based on SETLENGTH() behind #if
R_VERSION < R_Version(4, 3, 0) to avoid requiring your users to upgrade
from R >= 3.0.

A work-in-progress implementation of shrinkable vectors for data.table
can be found at [2]. The real problem is not implementing the same
simple API using SETLENGTH() and ALTREP, but in refactoring the rest of
the code not to violate its assumptions. (Total control over the
allocation of your shrinkable vectors is required: nothing good will
happen if the code tries to call ALTREP methods on an object that is
not of the exact ALTREP class it needs to be.)

My own notes on the use of ALTREP can be found at [3]. I will do my
best to keep them correct, but patches are always welcome.

And now a question:

Would R benefit from a patch to make xlengthgets() more like
EnlargeVector() [4] and sometimes return vectors with GROWABLE_BIT set?
(EnlargeVector() is currently only reachable from the `[<-` operation.)
It wouldn't be a deviation from the currently documented behaviour if
the function tested for NO_REFERENCES() and then used SETLENGTH() when,
say, reducing the length of a vector to 1/2 of its length or more.

-- 
Best regards,
Ivan

[1]
https://rdatatable-community.github.io/The-Raft/posts/2025-01-13-non-api-use/index.html#growable-vectors

[2]
https://github.com/Rdatatable/data.table/blob/growable_refactor/src/growable.c
Everyone is welcome to use this under the terms of GPL-2 or a later
version, or MPL-2, at their choice.

[3]
https://aitap.codeberg.page/R-api/#ALTREP

[4]
https://github.com/r-devel/r-svn/blob/59df414b844eea27c09b352ad30fd82e66764d2d/src/main/subassign.c#L256-L260