[Rd] Using a custom memory allocation function in R

[Simon Urbanek]

SO,

On Oct 22, 2014, at 10:53 AM, SO User <user3855417 at gmail.com> wrote:
> 
> Heads up: I posted this question on Stackoverflow yesterday
> (http://stackoverflow.com/questions/26484103/using-a-custom-memory-allocation-function-in-r).
> So far I haven’t gotten a response and I thought this could be an even
> better place to ask such a question.
> 
> I would like to be able to use my own memory allocation function for
> certain data structures (real valued vectors and arrays) in R. The
> reason for this is that I need my data to be 64bit aligned and I would
> like to use the numa library for having control over which memory node
> is used (I'm working on compute nodes with four 12-core AMD Opteron
> 6174 CPUs).
> 
> Now I have two functions for allocating and freeing memory:
> numa_alloc_onnode and numa_free(courtesy of
> http://stackoverflow.com/questions/8154162/numa-aware-cache-aligned-memory-allocation).
> I'm using R version 3.1.1, so I have access to the function
> allocVector3(src/main/memory.c), which seems to me as the intended way
> of adding a custom memory allocator. I also found the struct
> R_allocator in src/include/R_ext
> 
> However it is not clear to me how to put these pieces together. Let's
> say, in R, I want the result res of an evaluation such as
> 
> res <- Y - mean(Y)
> 
> to be saved in a memory area allocated with my own function, how would
> I do this? Can I integrate allocVector3 directly at the R level? I
> assume I have to go through the R-C interface. As far as I know, I
> cannot just return a pointer to the allocated area, but have to pass
> the result as an argument. So in R I call something like
> 
> n <- length(Y)
> res <- numeric(length=1)
> .Call("R_allocate_using_myalloc", n, res)
> res <- Y - mean(Y)
> 

That obviously won't work, because you're replacing your "res" with another binding - effectively removing anything you did. You could use something like

res <- .Call("R_allocate_using_myalloc", Y - mean(y))

if you wanted to convert "regular" R object into your own backed store.

However, the intent is typically to create objects with special backing store in your C code. One example are mmaped R objects - see
https://gist.github.com/s-u/6712c97ca74181f5a1a5
for an example.


> and in C
> 
> #include <R.h>
> #include <Rinternals.h>
> #include <numa.h>
> 

You are missing

#include <R_ext/Rallocators.h>



> SEXP R_allocate_using_myalloc(SEXP R_n, SEXP R_res){
> 
>  PROTECT(R_n = coerceVector(R_n, INTSXP));
>  PROTECT(R_res = coerceVector(R_res, REALSXP));
>  int *restrict n = INTEGER(R_n);
> 
>  R_allocator_t myAllocator;
>  myAllocator.mem_alloc = numa_alloc_onnode;
>  myAllocator.mem_free = numa_free;
>  myAllocator.res = NULL;
>  myAllocator.data = ???;
> 
>  R_res = allocVector3(REALSXP, n, myAllocator);
> 
>  UNPROTECT(2);
>  return R_res;
> }
> 
> Unfortunately I cannot get beyond a variable has incomplete type
> 'R_allocator_t' compilation error

Missed include above.


> (I had to remove the .data line
> since I have no clue as to what I should put there).

data is an opaque pointer that you can use for anything you want - R doesn't use it, it just passes it through so you can have auxiliary information attached to your allocation.


> Also the function
> signature for numa_alloc_onnode is (size_t size, int node). How do I
> pass the number of the memory node to the numa_alloc_onnode? Is that
> somehow done through the .data field?

Yes, you can supply it from your initiator function (just like the filename is passed in the mmap example).


> Does any of the above code make
> sense? Is there an easier way of achieving what I want to? It seems a
> bit odd to have to allocate a small vector in R and the change its
> location in memory in C just to be able to both control the memory
> allocation and have the vector available in R...
> 
> I'm trying to avoid using Rcpp, as I'm modifying a fairly large
> package and do not want to convert all C calls and thought that mixing
> different C interfaces could perform sub-optimally.
> 

AFAICT Rcpp wouldn't help you there - since it still has to go back to R for allocation of R objects.

Cheers,
Simon


> Any help is greatly appreciated.
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel