[R-pkg-devel] Replacement for SETLENGTH

[Merlise Clyde, Ph.D.]

Thanks for the added explanation Iris and Tomas!

So looking at the code for xlengthgets, it does appear that I may take a memory hit for multiple large objects due to the second allocation before the old objects are possibly garbage collected.     There are about 12 such instances per function that are returned (I do use a counter for keeping track of the number of PROTECTED and to UNPROTECT for bookkeeping :-).   For memory limited machines, the alloc/copy was a problem for memory usage - and if I recall was one of the reasons in 2008 I switched to SETLENGTH, which doesn't seem to do an allocation ???  If there is going to be an absolute ban on SETLENGTH  in packages I'll probably need to address memory management differently for those cases.

I did see a note before the function def'n of xlengthgets:

/* (if it is vectorizable). We could probably be fairly */
/* clever with memory here if we wanted to. */

It would seem that memcpy would be more efficient for at least some of the types  (REALSPX, INTSPX) unless I am missing something - but any way to be more clever with VECSPX ?

best,
Merlise



Merlise Clyde (she/her/hers)
Professor of Statistical Science and Director of Graduate Studies
Duke University

________________________________________
From: Iris Simmons <ikwsimmo using gmail.com>
Sent: Wednesday, January 15, 2025 1:00 AM
To: Merlise Clyde, Ph.D. <clyde using duke.edu>
Cc: r-package-devel using r-project.org <r-package-devel using r-project.org>
Subject: Re: [R-pkg-devel] Replacement for SETLENGTH
 
Hi Merlise!


Referring to here:

https://github.com/wch/r-source/blob/bb5a829466f77a3e1d03541747d149d65e900f2b/src/main/builtin.c#L834

It seems as though the object is only re-used if the new length is
equal to the old length.

If you use Rf_lengthgets, you will need to protect the return value.
The code you wrote that uses protect indexes looks correct, and the
reprotect is good because you no longer need the old object.

2 is the correct amount to unprotect. PROTECT and PROTECT_WITH_INDEX
(as far as I know) are the only functions that increase the size of
the protect stack, and so the only calls that need to be unprotected.
Typically, people define `int nprotect = 0;` at the start of their
functions, add `nprotect++;` after each PROTECT and PROTECT_WITH_INDEX
call, and add `UNPROTECT(nprotect);` immediately before each return or
function end. That makes it easier to keep track.

I typically use R_PreserveObject and R_ReleaseObject to protect
objects without a need to bind them somewhere in my package's
namespace. This would be that .onLoad() uses R_PreserveObject to
protect some objects, and .onUnload uses R_ReleaseObject to release
the protected objects. I probably would not use that for what you're
describing.


Regards,
    Iris

On Tue, Jan 14, 2025 at 11:26 PM Merlise Clyde, Ph.D. <clyde using duke.edu> wrote:
>
> I am trying to determine the best way to eliminate the use of SETLENGTH to truncate over allocated vectors in my package BAS to eliminate the NOTES about non-API calls in anticipation of R 4.5.0.
>
> From WRE:  "At times it can be useful to allocate a larger initial result vector and resize it to a shorter length if that is sufficient. The functions Rf_lengthgets and Rf_xlengthgets accomplish this; they are analogous to using length(x) <- n in R. Typically these functions return a freshly allocated object, but in some cases they may re-use the supplied object."
>
> it looks like using
>
>     x = Rf_lengthgets(x, newsize);
>     SET_VECTOR_ELT(result, 0, x);
>
> before returning works to resize without a performance hit that incurs with a copy.  (will this always re-use the supplied object if newsize < old size?)
>
> There is no mention in section 5.9.2 about the need for re-protection of the object,  but it seems to be mentioned in some packages as well as a really old thread about SET_LENGTH that looks like a  non-API MACRO to lengthgets,
>
> indeed if I call gc() and then rerun my test I have had some non-reproducible aborts in R Studio on my M3 Mac (caught once in R -d lldb)
>
> Do I need to do something more like
>
> PROTECT_INDEX ipx0;.
> PROTECT_WITH_INDEX(x0 = allocVector(REALSXP, old_size), &ipx0);
>
> PROTECT_INDEX ipx1;.
> PROTECT_WITH_INDEX(x1 = allocVector(REALSXP, old_size), &ipx1);
>
> # fill in values in x0 and  x1up to new_size (random) < old_size
> ...
> REPROTECT(x0 = Rf_lengthgets(x0, new_size), ipx0);
> REPROTECT(x1 = Rf_lengthgets(x1, new_size), ipx1);
>
> SET_VECTOR_ELT(result, 0, x0);
> SET_VECTOR_ELT(result, 1, x1);
> ...
> UNPROTECT(2);   # or is this 4?
> return(result);
>
>
> There is also a mention in WRE of R_PreserveObject and R_ReleaseObject -
>
> looking for advice if this is needed, or which approach is better/more stable to replace SETLENGTH?   (I have many many instances that need to be updated, so trying to get some clarity here before updating and running code through valgrind or other sanitizers to catch any memory issues before submitting an update to CRAN.
>
> best,
> Merlise
>
>
>
>
>
>
>
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-package-devel__;!!OToaGQ!ohDoxcAn5uIC25d42XhBz8Kd4YftOJDBoEW1NK9FOmgZpcmv0XIy5fQRm24-s_D8m9O_lR6jo6FcKiA$