[R-SIG-Finance] Update price data on disk using mmap package

[me]

Hi

if you  set

mmapPrices <- mmap(file=cFile, mode=double());

then mmapPrices is a ONE dimeansional array of double()'s. I think it
is easier to do the following, assuming you are working with a
univariate xts object. Define:

mmapPrices <- mmap(file=cFile,
struct(timestamp=double(),price=double()) )

Now assume that you have initialized mmapPrices with NA's, like this:

mmapPrices[,1] <-NA
mmapPrices[,2] <-NA

Now you can append a new xts row 'newbar' like so:

newbar <- xts(12.3, order.by=sometimestamp)

find the first free (timestamp==NA) row in mmapPrices

idxcol<-mmapPrices[,1]
iifree<-match(NA, idxcol[[1]] )

fill it with the data from newbar

mmapPrices[iifree,1] <- .index(newbar)
mmapPrices[iifree,2] <- coredata(newbar)[1,1]


This has been working fine for me.

Cheers

Soren

censix.com



On Wed, 29 Feb 2012 09:58:18 +0000
Wolfgang Wu <wobwu22 at gmail.com> wrote:

> Hello everyone,
> 
> I am currently storing my price data for fast access as a mmap file
> on disk. This works great and is super fast. All data is in one big
> file which I subset to get the data I need into memory.
> 
> Now I will need to update that data that is stored on disk on a daily 
> basis to store new data. The solution I currently have is to load all 
> data into memory, append the new data to the data in memory and then 
> store everything again as mmap on disk. Unfortunately I am running
> into memory problems now when loading all of the data into memory.
> 
> So my question is:  Is it possible to append new prices directly into
> my file on disk without loading everything into memory? And if, how?
> 
> Any help is appreciated!!
> 
> 
> Here is an example of what I am trying to do:
> 
> library(quantmod)
> library(mmap)
> 
> #Just for setting up the example
> getSymbols('SPY')
> xtsPrices <- SPY[1:(nrow(SPY)-1), ];
> xtsNewPrice <- last(SPY);
> 
> #Save prices in mmap on disk
> cFile <- 'C:/temp/PriceData.mmap'
> nPrices <- as.numeric(xtsPrices);
> mmapPrices <- as.mmap(nPrices, mode=double(), file=cFile);
> munmap(mmapPrices);
> 
> #Now I would like to append the new prices directly to the mmap
> object on disk
> mmapPrices <- mmap(file=cFile, mode=double());
> nNewPrice <- as.numeric(xtsNewPrice);
> mmapPrices <- c(mmapPrices, nNewPrice); #This line does not work how
> I hoped it would.
> 
> 
> 
> 
> Regards,
> 
> Wolfgang
> 
> _______________________________________________
> R-SIG-Finance at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R
> questions should go.