[Rd] Is it possible to increase MAX_NUM_DLLS in future R releases?

[Dan Dillon]

I have read both the historical r-devel threads and the most recent one
regarding this.  After reviewing the code, it would seem to beg for a
linked-list implementation vs. an array implementation.

This has several interesting consequences.

Cons:

1) We pay for dynamic allocation every time we register a DLL.  This isn't
a problem in the grand scheme of things as registering a DLL results in
other allocations (and is fairly infrequent).
2) Traversing the linked list is marginally more expensive than traversing
an array due to memory locality and prefetching.

Pros:

1) For the average R usage we decrease the memory footprint as we only
allocate what is needed.
2) For extreme use cases, we support it, but user beware (as symbol lookup
time will scale with number of loaded DLLs)
3) We open up the possibility of more dynamic ordering of the LoadedDLL
structure.  Essentially we can very cheaply move the most recently "used"
DLL to the front of the list.

The cons, I think, are small in this situation.  We already dynamically
allocate strings and arrays in the DllInfo strucutre so those are already
not memory-local.  Additionally, R is frequently operating on large
datasets so cache thrashing is almost a given.  The negligible time
increase in traversing the list, I believe, will be offset by the more
dynamic sorting that we are able to give that list.

I have written a patch that does this, however, I have not yet extensively
tested it.  If there is interest in including this in R for the future, I
am more than happy to do the testing necessary, however, if there is a
compelling reason to avoid this sort of implementation, I don't want to
spend hours testing.

The patch is available at
https://gist.github.com/dcdillon/814e769adbf53ff43961f106008b3312

Additionally, this will require cleanup at shutdown which I haven't yet
added, but will add it if it gets any traction.

Any feedback you have would be wonderful, and thank you for your time.

Dan Dillon

	[[alternative HTML version deleted]]