<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">Wiadomość napisana przez Vladimir Morozov <<a href="mailto:vmorozov2006@gmail.com" class="">vmorozov2006@gmail.com</a>> w dniu 06.09.2019, o godz. 20:04:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Daniel<div class="">Thanks a lot.</div><div class="">Those are very helpful ideas.</div><div class=""><br class=""></div><div class="">rbind_append --> it still has to allocate memory for the resulting series... so if memory allocation was the main reason for slow performance, maybe rbind_append doesn't change much? what do you think?</div><div class=""><br class=""></div><div class="">preallocating regular-interval time-series is a good idea.</div><div class="">however financial data are irregularly spaced (sometimes there may not be any price updates for a few secs, even more).</div><div class="">even if we postulate that prices are allowed to change no more than once per second, there's a lot of uses for the frequency of price updates, not only the values of the prices (the simplest assumption is the poisson arrival process for the updates, but there are many fancier, more powerful models...)</div><div class="">so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down many things!</div><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div>let's start with what exactly do you want to do? Do you want to collect market data and save it to disk? Or maybe you want to have a real-time strategy? These are two different problems and require distinct solutions.</div><div><br class=""></div><div>1) market data storage - why do you want to use R here? Isn't it better to dump the memory using mmap syscall and then import it into the database or R?</div><div><br class=""></div><div>2) real-time market strategy in R - in this case your lookback is limited. So if you add new data point, you can also discard/drop the oldest. In this way, your memory usage will remain at the same low level. If this solution suits you, then you can write a fast function in C here that would operate on the xts object.</div><div><br class=""></div><div>There is no such thing as matrix in R - this is a multidimensional vector. Let's say we have classic OHLC data for xts object:</div><div><br class=""></div><div>O H L C</div><div>O H L C</div><div>O H L C</div><div>O H L C</div><div>O H L C</div><div><br class=""></div><div><br class=""></div><div>In the memory of the data looks like one long vector.</div><div><br class=""></div><div>x: OOOOOHHHHHLLLLLCCCCC</div><div><br class=""></div><div>You can be clever here and use memcpy():</div><div><br class=""></div><div>memcpy(&xp + 1, &xp, (nrows(x) - 1)) * sizeof(double)); // or int - use: switch((TYPEOF(x))</div><div><br class=""></div><div>memcpy(&index_p + 1, &index_p, (nrows(x) - 1)) * sizeof(double)); // or int for Date() type</div><div><br class=""></div><div>This will move the memory so that the oldest value will be overwritten:</div><div><br class=""></div><div> 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5</div><div>x: OOOOH HHHHL LLLLC CCCC N</div><div><br class=""></div><div>Then you can add a new index and value.</div><div><br class=""></div><div>You will have preallocated memory at all times and you will use memory copy as little as possible. And the most important: you'll be operating on the xts object all time, so your code in R will be very fast :)</div><div><br class=""></div><div>It is advanced solutions - you need to understand not only how R's internals works, but also have a good C skills. If you want to use R for real-time trading, it's worth learn these things.</div><div><br class=""></div><div>Daniel</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="">i wish i could pre-allocate the vector for the values and maybe indices, but then do the assignment of the sort:</div><div class="">(say, in C++ i would have a method)</div><div class=""> price.set_next_point(time, value);<br class=""></div><div class=""><br class=""></div><div class="">thanks!</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <<a href="mailto:daniel.cegielka@gmail.com" class="">daniel.cegielka@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br class="">
<br class="">
> Wiadomość napisana przez Daniel Cegiełka <<a href="mailto:daniel.cegielka@gmail.com" target="_blank" class="">daniel.cegielka@gmail.com</a>> w dniu 06.09.2019, o godz. 16:10:<br class="">
> <br class="">
<br class="">
> <br class="">
> 2) preallocation<br class="">
> <br class="">
> preallocate_matrix <- function(n)<br class="">
> {<br class="">
> x <- matrix()<br class="">
> length(x) <- 4 * n # bid, ask, bid_size, ask_size<br class="">
> dim(x) <- c(n, 4) # see: ?dim<br class="">
> return(x)<br class="">
> }<br class="">
> <br class="">
> > x <- preallocate_matrix(5)<br class="">
> > x<br class="">
> [,1] [,2] [,3] [,4]<br class="">
> [1,] NA NA NA NA<br class="">
> [2,] NA NA NA NA<br class="">
> [3,] NA NA NA NA<br class="">
> [4,] NA NA NA NA<br class="">
> [5,] NA NA NA NA<br class="">
<br class="">
?matrix<br class="">
<br class="">
Usage<br class="">
matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,<br class="">
dimnames = NULL)<br class="">
<br class="">
so we don't even need preallocate_matrix() function<br class="">
<br class="">
> x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)<br class="">
> x<br class="">
[,1] [,2] [,3] [,4]<br class="">
2019-09-06 17:07:27 NA NA NA NA<br class="">
2019-09-06 17:07:28 NA NA NA NA<br class="">
2019-09-06 17:07:29 NA NA NA NA<br class="">
2019-09-06 17:07:30 NA NA NA NA<br class="">
2019-09-06 17:07:31 NA NA NA NA<br class="">
<br class="">
<br class="">
<br class="">
</blockquote></div>
</div></blockquote></div><br class=""></body></html>