This is about Jeff Ryan's mmap package (for which documentation is sparse).

My problem: I'm wondering what an effective way to *append* to an existing
mmapped file which is already "full of data" would be. e.g. simple case: if
the file contains 100 rows of data (as represented by mmap in R), I want to
add say another 5 rows of new data, giving a new total of 105 rows of data.
Surely many others have dealt with this problem before.  A common case
would be appending new tick data to a large file via mmap.

A quick search on SO gives a few discussions about remapping (mremap?) the
mmap in general, as a method for appending to an existing mmapped file, but
I haven't found that functionality to be available in the R mmap package.
 Before diving into the mmap 'src' code for possibly many more hours than I
would prefer to right now, I thought I'd first check by here and see if any
one can enlighten me on whether there is a clean/clever way to solve my
problem with R mmaps, which I may have over looked.

I started experimenting with trying to "extend" an existing mmap object in
R, but I got an odd result, as outlined in the example below (output of
results in comments in code).  I should point out that right now I don't
know if trying to append additional data/bytes to an existing map, like in
this example, is a good/safe idea in terms of memory usage.  In the simple
example below I tried setting up a file containing the values 1:5, with the
goal of appending a sixth element with the value of 9.  The result was
rather strange:

##########################################################################################
library(mmap)

filename = "pieceappend2"
writeBin(1:5,filename)

m <- mmap(filename, mode = int32())
m[]
#[1] 1 2 3 4 5
munmap(m)
rm(m)

numbytes <- sizeof(int32())
newelements <- 3 # add one new element, but show 3 new elements anyway

m2 <- mmap(filename, mode = int32(), len = numbytes * (5 + newelements))
m2[]
#[1] 1 2 3 4 5 0 0 0
m2[6] <- 9 # append data
m2[]
#[1] 1 2 3 4 5 9 0 0
munmap(m2)
rm(m2)

m3 <- mmap(filename, mode = int32())
m3[]
#[1] 1 2 3 4 5
# Does not show additional 9 0 0.
# Checking the file from the linux terminal with 'xxd pieceappend2' reveals
that "pieceappend2" has not changed (still contains 1:5); I expected file
to 1 2 3 4 5 9 0 0.
rm(m3)


m4 <- mmap(filename, mode = int32(), len = numbytes * (5 + newelements))
m4[]
#[1] 1 2 3 4 5 9 0 0
# How did m4 know the value of 9, where was this 9 stored??

#########################

It looked like when I set the len argument to len = numbytes * (5 +
newelements), and set m[6] <- 9, things seemed to have worked.  But then I
reloaded the map (assigned to m3) the original map length was returned.
 trying again with m4, the updated element is displayed!  But the weird
thing is, the original file, "pieceappend2" seems to only contain the
original data (1:5), which explains the behaviour of m3[] above.  But then
how did m4 know the extra appended element was still around?


One related approach/solution, but not quite the same, as shown very kindly
by the author of the code at http://censix.com/, is to update to a
preallocated mmap in a file with a very large number of rows prefilled with
NAs. But this approach requires making new files/"volumes" as the existing
files fill up.  Furthermore, I'm not quite sure if there is a clean way to
aggregate data for a given security which might be split across multiple
files.  For example, say I have tick data files for each month; one for
Jan, one for feb, one for march, etc.  Now suppose I want to pull into R
the tick data just for 29 May 2014 to 3 June 2014, which is split across 2
files?  I have to load into memory both files (or portions of them with
mmap), and merge the data.  It would seem it would just be easier to append
to one file to begin with, rather than split across files.

	[[alternative HTML version deleted]]

