[R-SIG-Finance] Parallelizing applyStrategy to multiple symbols
Brian G. Peterson
brian at braverock.com
Mon Mar 6 20:45:41 CET 2017
I suspect you're running up against communication and memory management
time and resource contention.
applyIndicators and applySignals should all be using vectorized code,
so the potential benefit from parallelization will likely be negative,
as communication and memory management swap any benefit from the
calculations.
applyRules might benefit from parllelization, but you would need to
come back together on any rebalancing period. You would also have
significant copying time.
If you were going to make this work, you'd need to minimize copies.
Your effective 'reduce' operation at the end by only returning
tradeStats could do this for the end of the calculation, but at the
start, you'd need to be smarter about how you segment market data to
each worker.
Just putting getSymbols on the workers might run into I/O contention
issues. You also don't need to redeclare the strategy object. You
could just copy that to each worker.
When we've done things as a one-off, we typically create portfolios for
each segment, and try to avoid as many copies as we can.
You'd need to profile to see exactly where you're getting hung up, but
this approach seems too simplistic (see my first sentence for hints).
We haven't bothered to do this in the package itself since with a
little work we can usually get to around one core minute per symbol per
day on L1 tick data, which means that even a large backtest on tick
data can finish in a few hours. The cost of optimizing execution time
doesn't seem to be worth the cost in programming and testing time.
Regards,
Brian
--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock
On Mon, 2017-03-06 at 18:53 +0000, Atakan Okan wrote:
> Hello,
>
> I am trying to parallelize applyStrategy() to make it faster when
> applied to multiple symbols. The reproducible code below only
> contains 3 symbols thus it finishes fast however when I apply it to
> 100 symbols in an index, sequential computing takes a lot of time.
> What is the best way to accomplish this? Using foreach loop does not
> seem to work and couldn't find any info on stackexchange or the usual
> mailing lists.
>
> Thanks.
>
> Atakan Okan
>
> Code with applyStrategy (foreach is below this):
>
> library(quantmod)
> library(quantstrat)
>
> symbols <- c("AAPL","GOOGL","MSFT")
>
> getSymbols(Symbols = symbols, from = "2010-01-01")
>
> currency('USD')
> stock(symbols, currency="USD")
>
> strategy.st <- "multiple_symbols_parallel_applystrategy"
> rm.strat(strategy.st)
>
>
> initPortf(strategy.st, symbols = symbols)
> initAcct(strategy.st, portfolios=strategy.st, initEq=100000)
> initOrders(portfolio=strategy.st)
> strategy(strategy.st,store=TRUE)
>
> rule.longenter = TRUE
> rule.longexit = TRUE
> rule.shortenter = TRUE
> rule.shortexit = TRUE
>
> txn.model <- 0
>
> add.indicator(strategy.st,
> name = "MACD",
> arguments = list(x=Cl(get(symbols))),
> label='macd')
>
> add.signal(strategy.st,name="sigCrossover",
> arguments = list(columns=c("macd.macd","signal.macd"),
> relationship="gt"),
> label="macd.gt.signal")
>
> add.signal(strategy.st,name="sigCrossover",
> arguments = list(columns=c("macd.macd","signal.macd"),
> relationship="lt"),
> label="macd.lt.signal")
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.gt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty= 1000,
> #osFUN="osAllInLong",
> ordertype='market',
> orderside='long',
> orderset='ocolong',
> TxnFees = txn.model),
> type='enter',
> label='longenter',
> enabled=FALSE
> )
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.lt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty='all',
> ordertype='market',
> orderside='long',
> orderset='ocolong',
> TxnFees = txn.model),
> type='exit',
> label='longexit',
> enabled=FALSE
> )
>
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.lt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty=-1000,
> #osFUN="osAllInShort",
> ordertype='market',
> orderside='short',
> orderset='ocoshort',
> TxnFees = txn.model),
> type='enter',
> label='shortenter',
> enabled=FALSE
> )
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.gt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty='all',
> ordertype='market',
> orderside='short',
> orderset='ocoshort',
> TxnFees = txn.model),
> type='exit',
> label='shortexit',
> enabled=FALSE
> )
>
> enable.rule(strategy.st,type="enter",label="longenter", enable =
> rule.longenter)
> enable.rule(strategy.st,type="exit",label="longexit", enable =
> rule.longexit)
> enable.rule(strategy.st,type="enter",label="shortenter", enable =
> rule.shortenter)
> enable.rule(strategy.st,type="exit",label="shortexit", enable =
> rule.shortexit)
> summary(getStrategy(strategy.st))
>
>
> applyStrategy( strategy=strategy.st ,
> portfolios=strategy.st,
> symbols = symbols,
> verbose=TRUE)
> updatePortf(strategy.st)
> updateAcct(strategy.st)
> updateEndEq(strategy.st)
>
> -------------------------------------------------------------------
> -------------------------------------------------------------------
> -------------------------------
> Code with foreach:
>
> library(quantmod)
> library(quantstrat)
>
> if(Sys.info()["sysname"] == "Windows") {
> library(doSNOW)
> cl <- makeCluster(4)
> registerDoSNOW(cl)
> }
> if(Sys.info()["sysname"] == "Linux") {
> library(doMC)
> registerDoMC(cores=4)
> #registerDoSEQ()
> getDoParWorkers()
> }
>
> symbols <- c("AAPL","GOOGL","MSFT")
>
> sens.df <- foreach(sym = 1:length(symbols),
> .combine = 'rbind',
> .packages = c("quantstrat","quantmod")) %dopar% {
>
> getSymbols(Symbols = sym, from = "2010-01-01")
>
> currency('USD')
> stock(sym, currency="USD")
>
> strategy.st <- "multiple_symbols_parallel_applystrategy"
>
> rm.strat(strategy.st)
>
>
> initPortf(strategy.st, symbols = sym)
> initAcct(strategy.st, portfolios=strategy.st, initEq=100000)
> initOrders(portfolio=strategy.st)
> strategy(strategy.st,store=TRUE)
>
> rule.longenter = TRUE
> rule.longexit = TRUE
> rule.shortenter = TRUE
> rule.shortexit = TRUE
>
> txn.model <- 0
>
> add.indicator(strategy.st,
> name = "MACD",
> arguments = list(x=Cl(get(sym))),
> label='macd')
>
> add.signal(strategy.st,name="sigCrossover",
> arguments = list(columns=c("macd.macd","signal.macd"),
> relationship="gt"),
> label="macd.gt.signal")
>
> add.signal(strategy.st,name="sigCrossover",
> arguments = list(columns=c("macd.macd","signal.macd"),
> relationship="lt"),
> label="macd.lt.signal")
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.gt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty= 1000,
> #osFUN="osAllInLong",
> ordertype='market',
> orderside='long',
> orderset='ocolong',
> TxnFees = txn.model),
> type='enter',
> label='longenter',
> enabled=FALSE
> )
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.lt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty='all',
> ordertype='market',
> orderside='long',
> orderset='ocolong',
> TxnFees = txn.model),
> type='exit',
> label='longexit',
> enabled=FALSE
> )
>
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.lt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty=-1000,
> #osFUN="osAllInShort",
> ordertype='market',
> orderside='short',
> orderset='ocoshort',
> TxnFees = txn.model),
> type='enter',
> label='shortenter',
> enabled=FALSE
> )
>
> add.rule(strategy.st,
> name='ruleSignal',
> arguments = list(sigcol="macd.gt.signal",
> sigval=TRUE,
> prefer="Open",
> orderqty='all',
> ordertype='market',
> orderside='short',
> orderset='ocoshort',
> TxnFees = txn.model),
> type='exit',
> label='shortexit',
> enabled=FALSE
> )
>
> enable.rule(strategy.st,type="enter",label="longenter", enable =
> rule.longenter)
> enable.rule(strategy.st,type="exit",label="longexit", enable =
> rule.longexit)
> enable.rule(strategy.st,type="enter",label="shortenter", enable =
> rule.shortenter)
> enable.rule(strategy.st,type="exit",label="shortexit", enable =
> rule.shortexit)
>
> summary(getStrategy(strategy.st))
>
>
> applyStrategy( strategy=strategy.st ,
> portfolios=strategy.st,
> symbols = sym,
> verbose=TRUE)
> updatePortf(strategy.st)
> updateAcct(strategy.st)
> updateEndEq(strategy.st)
>
> results.checkstrat <- data.frame(t(tradeStats(strategy.st)))
>
> return(results.checkstrat[,1])
>
> }
>
> if (Sys.info()["sysname"] == "Windows"){
> snow::stopCluster(cl) #dosnow windows
> }
>
> _______________________________________________
> R-SIG-Finance at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R
> questions should go.
More information about the R-SIG-Finance
mailing list