I do what Brian described, and I use a couple functions from FinancialInstrument to do it.<div><br><div>library(FinancialInstrument)</div><div>?saveSymbols.days</div><div>?saveSymbols.common</div><div>?getSymbols.FI<br><br>
(I just noticed that those 2 saveSymbols.* functions do not allow for a data extension other</div><div>than the old .rda. I will probably update that today.)<br><br>I put together a little example, which I'll attach as well as paste below.</div>
<div><br></div><div>This is how I do it, but I certainly encourage suggestions for improvement.</div><div><br></div><div>HTH,</div><div>Garrett<br><br><div>> library(FinancialInstrument)</div><div><br></div><div>> # object with daily periodicity</div>
<div>> data(sample_matrix)</div><div>> DDD <- as.xts(sample_matrix)</div><div><br></div><div>> #object with minute periodicity</div><div>> AAA <- xts(rnorm(1:10000), Sys.time()-(60*1:10000))</div><div>> AAA <- align.time(AAA)</div>
<div>> colnames(AAA) <- "AAA"</div><div><br></div><div>> # look at the objects we're going to store</div><div>> head(AAA)</div><div> AAA</div><div>2011-10-31 09:04:00 0.05152989</div>
<div>2011-10-31 09:05:00 0.12797379</div><div>2011-10-31 09:06:00 0.96025183</div><div>2011-10-31 09:07:00 -0.23265907</div><div>2011-10-31 09:08:00 1.77706849</div><div>2011-10-31 09:09:00 -1.29139344</div><div><br></div>
<div>> head(DDD)</div><div> Open High Low Close</div><div>2007-01-02 50.03978 50.11778 49.95041 50.11778</div><div>2007-01-03 50.23050 50.42188 50.23050 50.39767</div><div>2007-01-04 50.42096 50.42096 50.26414 50.33236</div>
<div>2007-01-05 50.37347 50.37347 50.22103 50.33459</div><div>2007-01-06 50.24433 50.24433 50.11121 50.18112</div><div>2007-01-07 50.13211 50.21561 49.99185 49.99185</div><div><br></div><div>> mydir <- getwd()</div>
<div>> saveSymbols.days("AAA", base_dir=mydir)</div><div>> saveSymbols.common("DDD", base_dir=mydir)</div><div><br></div><div>> # now that they are on disk,</div><div>> # remove them from workspace</div>
<div>> rm("AAA", "DDD")</div><div><br></div><div>> # get from disk</div><div>> getSymbols("AAA", src='FI', dir=mydir, split_method='days', from='2011-10-31')</div>
<div>[1] "AAA"</div><div><br></div><div>> getSymbols("DDD", src='FI', dir=mydir, split_method='common')</div><div>[1] "DDD"</div><div><br></div><div>> head(AAA)</div><div>
AAA</div><div>2011-10-31 09:04:00 0.05152989</div><div>2011-10-31 09:05:00 0.12797379</div><div>2011-10-31 09:06:00 0.96025183</div><div>2011-10-31 09:07:00 -0.23265907</div><div>2011-10-31 09:08:00 1.77706849</div>
<div>2011-10-31 09:09:00 -1.29139344</div><div><br></div><div>> head(DDD)</div><div> Open High Low Close</div><div>2007-01-02 50.03978 50.11778 49.95041 50.11778</div><div>2007-01-03 50.23050 50.42188 50.23050 50.39767</div>
<div>2007-01-04 50.42096 50.42096 50.26414 50.33236</div><div>2007-01-05 50.37347 50.37347 50.22103 50.33459</div><div>2007-01-06 50.24433 50.24433 50.11121 50.18112</div><div>2007-01-07 50.13211 50.21561 49.99185 49.99185</div>
<div><br></div><div>> #--------</div><div>> # You can setSymbolLookup so that getSymbols will know where</div><div>> # to look. There are 2 ways to setSymbolLookup: explicitly, </div><div>> # or by setting the "src" field of an instrument.</div>
<div>> </div><div>> # explicitly</div><div>> setSymbolLookup(DDD=list(src='FI', dir=mydir, split_method='common'))</div><div><br></div><div>> getSymbols("DDD")</div><div>[1] "DDD"</div>
<div><br></div><div>> # by using the "src" field of an instrument</div><div>> stock("AAA", currency("USD"), src=list(src='FI', dir=mydir, split_method='days'))</div><div>
[1] "AAA"</div><div><br></div><div>> getSymbols("AAA", from='2011-10-31')</div><div>[1] "AAA"</div><div><br></div><div>> # cleanup</div><div>> rm("AAA", "DDD")</div>
<div>> unlink("AAA", recursive=TRUE)</div><div>> unlink("DDD", recursive=TRUE)</div><div><br></div><br><br><div class="gmail_quote">On Mon, Nov 7, 2011 at 3:20 AM, Brian G. Peterson <span dir="ltr"><<a href="mailto:brian@braverock.com">brian@braverock.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Sun, 2011-11-06 at 22:43 -0500, Dino Veritas wrote:<br>
> Hello, I recently found this list and have been reading deeply the<br>
> archives. I am wondering how people here maintain their collections of data<br>
> for easy use in R. I am wondering a few things:<br>
><br>
> 1) How do members of this list deal with keeping persistent data<br>
> collections with R? I was thinking of individual xts objects by asset and<br>
> frequency (such as AAPL daily, AAPL minute, AAPL 60m, etc). While I can<br>
> store and maintain these xts objects on disk and load them into R as<br>
> needed, I am wondering if there is a more better solution.<br>
<br>
</div>I store only tick data, as I can easily get to any other frequency from<br>
tick. I've considered also storing daily data, but in the end I decide<br>
it is too much trouble to (additionally) manage, and just store tick.<br>
<div class="im"><br>
> 2) Coming from that, I have been looking into the indexing package for my<br>
> needs. It seems very useful for managing a lot of large data sets in<br>
> memory, but I am not sure it is a good method for maintaining persistent<br>
> data, I have found trouble adding information to existing data that is<br>
> indexed on disk. Do poster here use indexing for this purpose? I did find<br>
> an old post or two touching on that with no specifics. I would like to be<br>
> able to combine the ability of indexing to have many large data sets<br>
> available in memory with persistent storage of data. Has anyone any<br>
> experience doing this?<br>
<br>
</div>You are correct that the 'indexing' package is very powerful. It is<br>
also not done yet.<br>
<br>
As I said, I store tick data. The way I do this is with single files<br>
per day of data per symbol, pre-parsed into xts objects and stored to<br>
disk in one directory per symbol (using 'save').<br>
<br>
I then use FinancialInstrument to keep track of all the instrument<br>
metadata, and getSymbols to load the data into R when I need it (and<br>
over the time-frames that I require). We currently download tick data<br>
for about 2500 tradeable instruments per day, and maintain archives<br>
going back several years. We have the .instrument environment stored on<br>
the same file server as the data, and every .Rprofile in the firm points<br>
to this so that everyone has access to getInstrument and getSymbols<br>
<br>
I know someone who works in the hedge fund industry, mostly with monthly<br>
data, with some daily data sprinkled in. He uses the same approach I<br>
have outlined of storing the metadata in FinancialInstrument, and<br>
getSymbols to access the data. He typically stores one consolidated CSV<br>
file per instrument, because CSV files are easy to add on to with a<br>
batch process.<br>
<br>
For lower frequency data (let's say daily or lower) a database is<br>
certainly an option, and there are getSymbols wrappers that could be<br>
adapted to whatever schema you decided to use. Obviously, there are tick<br>
data database providers such as OneTick and kdb, and if you have this<br>
problem and the resources to need this type of solution, you probably<br>
already know that you are in this camp, and know that these providers<br>
have R interfaces of varying quality.<br>
<br>
The FinancialInstrument package has a 'parsers' directory included in<br>
the 'inst' directory of the package with many examples of download and<br>
parse routines for regular loading of data from a variety of free or<br>
subscription providers. This should give you a lot of material to begin<br>
working with your own data providers.<br>
<div class="im"><br>
> 3) How do people keep track of all the data sets within R? Are there any<br>
> useful packages for keeping track of multiple sets of financial data and<br>
> the information about them?<br>
<br>
</div>We wrote and use FinancialInstrument for this purpose.<br>
<br>
As I said earlier, I see no value in storing different periodicities,<br>
and store only tick.<br>
<br>
One of the reasons that I chose to write a getSymbols wrapper for<br>
retrieving our tick data stores is that resources like this list have<br>
extensive experience about using getSymbols, and it is therefore easy<br>
for people at our firm to become familiar with using the data.<br>
<br>
Also, I am reasonably confident that as the indexing package matures,<br>
there will be a getSymbols method for it as well, and if appropriate I<br>
can easily convert all my data in one batch pass and it will be<br>
transparent to my users.<br>
<br>
I made what I now realize to have been a mistake at a previous firm in<br>
writing a data retrieval function that was not compatible with<br>
getSymbols which was more complex to teach people how to use it, and<br>
less compatible with huge amounts of other publicly available code.<br>
<br>
quantmod and FinancialInstrument contain examples of various getSymbols<br>
methods that may meet your needs, or that could serve as templates for<br>
your custom in-house data source.<br>
<div class="im"><br>
> 4) Any other pointers? I know many here are well versed and manage large<br>
> data sets with R. Any tips you have or even simply showing me in a helpful<br>
> direction to useful packages you use is great. This list is a great help<br>
> for me and I am still browsing old threads!<br>
<br>
</div>Regards,<br>
<br>
- Brian<br>
<font color="#888888"><br>
--<br>
Brian G. Peterson<br>
<a href="http://braverock.com/brian/" target="_blank">http://braverock.com/brian/</a><br>
Ph: <a href="tel:773-459-4973" value="+17734594973">773-459-4973</a><br>
IM: bgpbraverock<br>
</font><div><div></div><div class="h5"><br>
_______________________________________________<br>
<a href="mailto:R-SIG-Finance@r-project.org">R-SIG-Finance@r-project.org</a> mailing list<br>
<a href="https://stat.ethz.ch/mailman/listinfo/r-sig-finance" target="_blank">https://stat.ethz.ch/mailman/listinfo/r-sig-finance</a><br>
-- Subscriber-posting only. If you want to post, subscribe first.<br>
-- Also note that this is not the r-help list where general R questions should go.<br>
</div></div></blockquote></div><br></div></div>