[R-SIG-Finance] Parallelizing applyStrategy to multiple symbols
Atakan Okan
atakanokan at outlook.com
Mon Mar 6 23:36:32 CET 2017
Hi Frank,
I just thought of an idea based on your suggestion. Instead of trying to implement a foreach loop, I will try to subset my symbol set into different R sessions with the create a new r session option in Rstudio and then run each subset on a different session with the default call to applyStrategy. I think this is what you were suggesting or I might have understood it incorrectly.
Hi Brian,
My understanding of parallelization wasnt enough to grasp all of your reply, but I am not planning on doing rebalancing or testing any strategy that need to "talk" to other threads. Each symbol is backtested on its own withiut any input or output to and from other symbols' backtest. Would my idea suggested above work in this case? I think I explained my problem inadequately; the time of completion of a single symbol's backtest is not the issue but the sequential computing of each symbol's backtest and consequently, linearly increasing completion time of all symbols' backtest is the main issue. I just want to divide each symbol's applyStrategy call to each CPU my laptop has to speed up the process. Like apply.paramset but not for each parameter combination, for each symbol. I hope I have explained better.
Thanks for the help.
Best,
Atakan Okan
> On 6 Mar 2017, at 23:55, Frank <frankm60606 at gmail.com> wrote:
>
> Atakan,
>
> What kind of computer do you have? Number of cores, memory, hyperthreaded or not?
>
> /Brian Does this package take advantage of hyperthreading? By your comment it suggests it does for multiple cores and I would assume hyper threading.
>
> When I do non-R computer intensive work, I break it up into chunks of 8. I have an i7 that hyper threads which pegs the CPU at 100%. If you had a similar setup, you could break your 100 symbol list down into 8 datasets and run them simultaneously.
>
> Regardless, adding memory is usually a cheap and mindless way to improve throughput.
>
> Best,
>
> Frank
> Chicago, IL
>
> -----Original Message-----
> From: R-SIG-Finance [mailto:r-sig-finance-bounces at r-project.org] On Behalf Of Brian G. Peterson
> Sent: Monday, March 06, 2017 1:46 PM
> To: Atakan Okan <atakanokan at outlook.com>; r-sig-finance at r-project.org
> Subject: Re: [R-SIG-Finance] Parallelizing applyStrategy to multiple symbols
>
> I suspect you're running up against communication and memory management time and resource contention.
>
> applyIndicators and applySignals should all be using vectorized code, so the potential benefit from parallelization will likely be negative, as communication and memory management swap any benefit from the calculations.
>
> applyRules might benefit from parllelization, but you would need to come back together on any rebalancing period. You would also have significant copying time.
>
> If you were going to make this work, you'd need to minimize copies.
> Your effective 'reduce' operation at the end by only returning tradeStats could do this for the end of the calculation, but at the start, you'd need to be smarter about how you segment market data to each worker.
>
> Just putting getSymbols on the workers might run into I/O contention issues. You also don't need to redeclare the strategy object. You could just copy that to each worker.
>
> When we've done things as a one-off, we typically create portfolios for each segment, and try to avoid as many copies as we can.
>
> You'd need to profile to see exactly where you're getting hung up, but this approach seems too simplistic (see my first sentence for hints).
>
> We haven't bothered to do this in the package itself since with a little work we can usually get to around one core minute per symbol per day on L1 tick data, which means that even a large backtest on tick data can finish in a few hours. The cost of optimizing execution time doesn't seem to be worth the cost in programming and testing time.
>
> Regards,
>
> Brian
>
> --
> Brian G. Peterson
> http://braverock.com/brian/
> Ph: 773-459-4973
> IM: bgpbraverock
>
>> On Mon, 2017-03-06 at 18:53 +0000, Atakan Okan wrote:
>> Hello,
>>
>> I am trying to parallelize applyStrategy() to make it faster when
>> applied to multiple symbols. The reproducible code below only contains
>> 3 symbols thus it finishes fast however when I apply it to
>> 100 symbols in an index, sequential computing takes a lot of time.
>> What is the best way to accomplish this? Using foreach loop does not
>> seem to work and couldn't find any info on stackexchange or the usual
>> mailing lists.
>>
>> Thanks.
>>
>> Atakan Okan
>>
>> Code with applyStrategy (foreach is below this):
>>
>> library(quantmod)
>> library(quantstrat)
>>
>> symbols <- c("AAPL","GOOGL","MSFT")
>>
>> getSymbols(Symbols = symbols, from = "2010-01-01")
>>
>> currency('USD')
>> stock(symbols, currency="USD")
>>
>> strategy.st <- "multiple_symbols_parallel_applystrategy"
>> rm.strat(strategy.st)
>>
>>
>> initPortf(strategy.st, symbols = symbols) initAcct(strategy.st,
>> portfolios=strategy.st, initEq=100000)
>> initOrders(portfolio=strategy.st)
>> strategy(strategy.st,store=TRUE)
>>
>> rule.longenter = TRUE
>> rule.longexit = TRUE
>> rule.shortenter = TRUE
>> rule.shortexit = TRUE
>>
>> txn.model <- 0
>>
>> add.indicator(strategy.st,
>> name = "MACD",
>> arguments = list(x=Cl(get(symbols))),
>> label='macd')
>>
>> add.signal(strategy.st,name="sigCrossover",
>> arguments = list(columns=c("macd.macd","signal.macd"),
>> relationship="gt"),
>> label="macd.gt.signal")
>>
>> add.signal(strategy.st,name="sigCrossover",
>> arguments = list(columns=c("macd.macd","signal.macd"),
>> relationship="lt"),
>> label="macd.lt.signal")
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.gt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty= 1000,
>> #osFUN="osAllInLong",
>> ordertype='market',
>> orderside='long',
>> orderset='ocolong',
>> TxnFees = txn.model),
>> type='enter',
>> label='longenter',
>> enabled=FALSE
>> )
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.lt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty='all',
>> ordertype='market',
>> orderside='long',
>> orderset='ocolong',
>> TxnFees = txn.model),
>> type='exit',
>> label='longexit',
>> enabled=FALSE
>> )
>>
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.lt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty=-1000,
>> #osFUN="osAllInShort",
>> ordertype='market',
>> orderside='short',
>> orderset='ocoshort',
>> TxnFees = txn.model),
>> type='enter',
>> label='shortenter',
>> enabled=FALSE
>> )
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.gt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty='all',
>> ordertype='market',
>> orderside='short',
>> orderset='ocoshort',
>> TxnFees = txn.model),
>> type='exit',
>> label='shortexit',
>> enabled=FALSE
>> )
>>
>> enable.rule(strategy.st,type="enter",label="longenter", enable =
>> rule.longenter)
>> enable.rule(strategy.st,type="exit",label="longexit", enable =
>> rule.longexit)
>> enable.rule(strategy.st,type="enter",label="shortenter", enable =
>> rule.shortenter)
>> enable.rule(strategy.st,type="exit",label="shortexit", enable =
>> rule.shortexit)
>> summary(getStrategy(strategy.st))
>>
>>
>> applyStrategy( strategy=strategy.st ,
>> portfolios=strategy.st,
>> symbols = symbols,
>> verbose=TRUE)
>> updatePortf(strategy.st)
>> updateAcct(strategy.st)
>> updateEndEq(strategy.st)
>>
>> -------------------------------------------------------------------
>> -------------------------------------------------------------------
>> -------------------------------
>> Code with foreach:
>>
>> library(quantmod)
>> library(quantstrat)
>>
>> if(Sys.info()["sysname"] == "Windows") {
>> library(doSNOW)
>> cl <- makeCluster(4)
>> registerDoSNOW(cl)
>> }
>> if(Sys.info()["sysname"] == "Linux") {
>> library(doMC)
>> registerDoMC(cores=4)
>> #registerDoSEQ()
>> getDoParWorkers()
>> }
>>
>> symbols <- c("AAPL","GOOGL","MSFT")
>>
>> sens.df <- foreach(sym = 1:length(symbols),
>> .combine = 'rbind',
>> .packages = c("quantstrat","quantmod")) %dopar% {
>>
>> getSymbols(Symbols = sym, from = "2010-01-01")
>>
>> currency('USD')
>> stock(sym, currency="USD")
>>
>> strategy.st <- "multiple_symbols_parallel_applystrategy"
>>
>> rm.strat(strategy.st)
>>
>>
>> initPortf(strategy.st, symbols = sym)
>> initAcct(strategy.st, portfolios=strategy.st, initEq=100000)
>> initOrders(portfolio=strategy.st)
>> strategy(strategy.st,store=TRUE)
>>
>> rule.longenter = TRUE
>> rule.longexit = TRUE
>> rule.shortenter = TRUE
>> rule.shortexit = TRUE
>>
>> txn.model <- 0
>>
>> add.indicator(strategy.st,
>> name = "MACD",
>> arguments = list(x=Cl(get(sym))),
>> label='macd')
>>
>> add.signal(strategy.st,name="sigCrossover",
>> arguments = list(columns=c("macd.macd","signal.macd"),
>> relationship="gt"),
>> label="macd.gt.signal")
>>
>> add.signal(strategy.st,name="sigCrossover",
>> arguments = list(columns=c("macd.macd","signal.macd"),
>> relationship="lt"),
>> label="macd.lt.signal")
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.gt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty= 1000,
>> #osFUN="osAllInLong",
>> ordertype='market',
>> orderside='long',
>> orderset='ocolong',
>> TxnFees = txn.model),
>> type='enter',
>> label='longenter',
>> enabled=FALSE
>> )
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.lt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty='all',
>> ordertype='market',
>> orderside='long',
>> orderset='ocolong',
>> TxnFees = txn.model),
>> type='exit',
>> label='longexit',
>> enabled=FALSE
>> )
>>
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.lt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty=-1000,
>> #osFUN="osAllInShort",
>> ordertype='market',
>> orderside='short',
>> orderset='ocoshort',
>> TxnFees = txn.model),
>> type='enter',
>> label='shortenter',
>> enabled=FALSE
>> )
>>
>> add.rule(strategy.st,
>> name='ruleSignal',
>> arguments = list(sigcol="macd.gt.signal",
>> sigval=TRUE,
>> prefer="Open",
>> orderqty='all',
>> ordertype='market',
>> orderside='short',
>> orderset='ocoshort',
>> TxnFees = txn.model),
>> type='exit',
>> label='shortexit',
>> enabled=FALSE
>> )
>>
>> enable.rule(strategy.st,type="enter",label="longenter", enable =
>> rule.longenter)
>> enable.rule(strategy.st,type="exit",label="longexit", enable =
>> rule.longexit)
>> enable.rule(strategy.st,type="enter",label="shortenter", enable =
>> rule.shortenter)
>> enable.rule(strategy.st,type="exit",label="shortexit", enable =
>> rule.shortexit)
>>
>> summary(getStrategy(strategy.st))
>>
>>
>> applyStrategy( strategy=strategy.st ,
>> portfolios=strategy.st,
>> symbols = sym,
>> verbose=TRUE)
>> updatePortf(strategy.st)
>> updateAcct(strategy.st)
>> updateEndEq(strategy.st)
>>
>> results.checkstrat <- data.frame(t(tradeStats(strategy.st)))
>>
>> return(results.checkstrat[,1])
>>
>> }
>>
>> if (Sys.info()["sysname"] == "Windows"){
>> snow::stopCluster(cl) #dosnow windows }
>>
>> _______________________________________________
>> R-SIG-Finance at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R
>> questions should go.
>
> _______________________________________________
> R-SIG-Finance at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>
More information about the R-SIG-Finance
mailing list