[R-SIG-Finance] Processing time of backtests on a single computer
david.jessop at ubs.com
david.jessop at ubs.com
Thu Apr 7 16:50:45 CEST 2016
Hi
Just to add to Brian's comment on "economic justification", see David Bailey / de Prado's sequence of papers (e.g. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2460551 or http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2308659 , the latter is probably better) on the perils of overfitting parameters. Summarising the paper, if you run just 100 backtests and optimise the parameters on a strategy with a true Sharpe ratio of zero then the best one will have on average an apparent Sharpe of 2.5.
Regards
David
-----Original Message-----
From: R-SIG-Finance [mailto:r-sig-finance-bounces at r-project.org] On Behalf Of Brian G. Peterson
Sent: 06 April 2016 21:49
To: Jersey Fanatic
Cc: r-sig-finance at r-project.org
Subject: Re: [R-SIG-Finance] Processing time of backtests on a single computer
You didn't say that you were doing parameter optimization.
The length of time that a parameter optimization using brute force will take is a linear combination of the number of parameter combinations that you choose to search.
Typically, you should only include parameter distributions for parameters for which you feel you have a strong economic justification.
Your strategy contains eight parameter distributions and two constraints. The choices of these distributions appears arbitrary, and will result in hundreds of combinations to test. So, you should expect your strategy to take approximately a linear combination of additional time based on the number of parameters you wish to test.
Another thing that will make this test take a long time is the inclusion of trailing stops. As is described in the documentation, trailing stops require evaluating the strategy at more points in the path dependent loop. The number of observations that need to be evaluated in the path dependent rules loop has another linear effect on the time to evaluate your backtest. Have you validated that there is a theoretical justification for a trailing stop? Does this increase the positive expectation of your resulting signal process?
Most of the validation of your indicator and signal processes should be possible long before you get to parameter optimization.
Regards,
Brian
--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock
On Wed, 2016-04-06 at 23:20 +0300, Jersey Fanatic wrote:
> The email with the CSV data attached is waiting for moderator's approval.
> The reproducible code is below.
>
>
> 2016-04-06 23:17 GMT+03:00 Jersey Fanatic <jerseyfanatic1 at gmail.com>:
>
> > Thanks for the quick reply. I did not understand what you meant by path
> > dependent looping or copying, but my aim is to decrease the processing time
> > of the backtesting of several combinations of parameters.
> >
> > Here is the reproducible example, and the data I am using is attached as
> > well:
> >
> > library(lattice)
> > library(foreach)
> > library(doSNOW)
> > library(ggplot2)
> > library(PerformanceAnalytics)
> > library(doSNOW)
> > require(latticeExtra)
> > require(grid)
> > library(gridExtra)
> > library(reshape)
> > library(quantstrat)
> > no.cores <- 8
> >
> > .strategy<- new.env()
> > .blotter<- new.env()
> >
> > currency(c('USD', 'EUR'))
> > exchange_rate(primary_id="EURUSD", tick_size=0.0001)
> >
> > data.location.r <- "processingtime_q_rsigfinance.csv"
> > symbol.data <- as.xts(read.zoo(data.location.r, sep=',',
> > tz="",header=TRUE, format='%d/%m/%Y %H:%M', index.column = 1))
> > symbol.data <- symbol.data[symbol.data$VOLUME!=0,]
> > # Delete rows with no volume (when the
> > market is closed)
> > symbol.data[,c(1,2,3,4)] <-
> > round(as.numeric(symbol.data[,c(1,2,3,4)]),abs(log10(0.0001))) #datayı tick
> > sizela uyumlu yapma
> > assign("EURUSD", symbol.data)
> >
> > strategy.st <- "rsigfinance"
> > rm.strat(strategy.st)
> >
> >
> > initDate = as.character(as.Date(index(symbol.data[1])-1))
> >
> > initPortf(strategy.st, "EURUSD", initDate=initDate, currency = "USD")
> > initAcct(strategy.st, portfolios=strategy.st, initDate=initDate,
> > initEq=100000, currency = "USD")
> > initOrders(portfolio=strategy.st,initDate=initDate)
> >
> > strategy(strategy.st,store=TRUE)
> > summary(getStrategy(strategy.st))
> >
> >
> > macdFastMARange <- seq(2,17,by=5)
> > macdSlowMARange <- seq(5,35,by=10)
> > macdSignalRange <- seq(2,18,by=8)
> >
> > StopLossDistanceRange <- seq(0.01,0.02,by=0.01)
> >
> > TrailingDistanceRange <- seq(0.01,0.02,by=0.01)
> >
> >
> > positionSizeLong = round(100000 / as.numeric(symbol.data$CLOSE[1]),-2)
> > positionSizeShort = - round(100000 / as.numeric(symbol.data$CLOSE[1]),-2)
> > txn.model <- 0
> > sltsltp.txn.fee <- 0
> >
> >
> > add.indicator(strategy.st,
> > name = "MACD",
> > arguments = list(x=Cl(eval(parse(text = "EURUSD")))),
> > label='macd')
> >
> > add.signal(strategy.st,name="sigCrossover",
> > arguments =
> > list(columns=c("macd.macd","signal.macd"),relationship="gt"),
> > label="macd.gt.signal")
> >
> > add.signal(strategy.st,name="sigCrossover",
> > arguments =
> > list(columns=c("macd.macd","signal.macd"),relationship="lt"),
> > label="macd.lt.signal")
> >
> > add.rule(strategy.st,
> > name='ruleSignal',
> > arguments = list(sigcol="macd.gt.signal",
> > sigval=TRUE,
> > prefer="Open",
> > orderqty= positionSizeLong,
> > ordertype='market',
> > orderside='long',
> > orderset='ocolong',
> > TxnFees = txn.model),
> > type='enter',
> > label='longenter',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st,
> > name='ruleSignal',
> > arguments = list(sigcol="macd.lt.signal",
> > sigval=TRUE,
> > prefer="Open",
> > orderqty='all',
> > ordertype='market',
> > orderside='long',
> > orderset='ocolong',
> > TxnFees = txn.model),
> > type='exit',
> > label='longexit',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st,name='ruleSignal',
> > arguments = list( sigcol="macd.lt.signal", sigval=TRUE,
> > replace=FALSE,
> > orderside='long',
> > ordertype='stoplimit',
> > tmult=TRUE,
> > threshold=quote( longStopLossDistance ),
> > orderqty='all',
> > orderset='ocolong',
> > TxnFees = txn.model),
> > type='chain', parent="longenter",
> > label='StopLossLong',
> > enabled=TRUE)
> >
> > add.rule(strategy.st, name = 'ruleSignal',
> > arguments=list(sigcol="macd.lt.signal" , sigval=TRUE,
> > replace=FALSE,
> > orderside='long',
> > ordertype='stoptrailing',
> > tmult=TRUE,
> > threshold=quote(longTrailingStopDistance),
> > orderqty='all',
> > orderset='ocolong',
> > TxnFees = txn.model),
> > type='chain', parent="longenter",
> > label='StopTrailingLong',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st, name = "ruleSignal",
> > arguments = list(sigcol="macd.lt.signal",
> > sigval=TRUE,
> > ordertype="limit",
> > orderside="long",
> > replace=FALSE,
> > tmult=TRUE,
> > threshold=quote(longTakeProfitDistance),
> > orderqty="all",
> > orderset="ocolong",
> > TxnFees = txn.model),
> > type = "chain", parent="longenter",
> > label = "takeProfitLong",
> > enabled = FALSE
> > )
> >
> > add.rule(strategy.st,
> > name='ruleSignal',
> > arguments = list(sigcol="macd.lt.signal",
> > sigval=TRUE,
> > prefer="Open",
> > orderqty=positionSizeShort,
> > ordertype='market',
> > orderside='short',
> > orderset='ocoshort',
> > TxnFees = txn.model),
> > type='enter',
> > label='shortenter',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st,
> > name='ruleSignal',
> > arguments = list(sigcol="macd.gt.signal",
> > sigval=TRUE,
> > prefer="Open",
> > orderqty='all',
> > ordertype='market',
> > orderside='short',
> > orderset='ocoshort',
> > TxnFees = txn.model),
> > type='exit',
> > label='shortexit',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st,name='ruleSignal',
> > arguments = list( sigcol="macd.gt.signal", sigval=TRUE,
> > replace=FALSE,
> > orderside='short',
> > ordertype='stoplimit',
> > tmult=TRUE,
> > threshold=quote( shortStopLossDistance ),
> > orderqty='all',
> > orderset='ocoshort',
> > TxnFees = txn.model),
> > type='chain', parent="shortenter",
> > label='StopLossShort',
> > enabled=TRUE)
> >
> > add.rule(strategy.st, name = 'ruleSignal',
> > arguments=list(sigcol="macd.gt.signal" , sigval=TRUE,
> > replace=FALSE,
> > orderside='short',
> > ordertype='stoptrailing',
> > tmult=TRUE,
> > threshold=quote( shortTrailingStopDistance),
> > orderqty='all',
> > orderset='ocoshort',
> > TxnFees = txn.model),
> > type='chain', parent="shortenter",
> > label='StopTrailingShort',
> > enabled=TRUE
> > )
> >
> > add.rule(strategy.st, name = "ruleSignal",
> > arguments = list(sigcol="macd.gt.signal",
> > sigval=TRUE,
> > ordertype="limit",
> > orderside="short",
> > replace=FALSE,
> > tmult=TRUE,
> > threshold=quote( -shortTakeProfitDistance),
> > orderqty="all",
> > orderset="ocoshort",
> > TxnFees = txn.model),
> > type = "chain", parent="shortenter",
> > label = "takeProfitShort",
> > enabled = FALSE
> > )
> >
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = 'indicator',
> > component.label = "macd",
> > variable = list( nFast = macdFastMARange ),
> > label = "macdFastMARANGE")
> >
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = 'indicator',
> > component.label = "macd",
> > variable = list( nSlow = macdSlowMARange ),
> > label = "macdSlowMARANGE")
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = 'indicator',
> > component.label = "macd",
> > variable = list( nSig = macdSignalRange ),
> > label = "macdSignalRANGE")
> >
> > add.distribution.constraint(strategy.st,
> > paramset.label = 'MACD_OPT',
> > distribution.label.1 = 'macdFastMARANGE',
> > distribution.label.2 = 'macdSlowMARANGE',
> > operator = '<',
> > label = 'FastMA<SlowMA')
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = "chain",
> > component.label = "StopLossLong",
> > variable = list( threshold = StopLossDistanceRange ),
> > label = "StopLossLONG")
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = "chain",
> > component.label = "StopTrailingLong",
> > variable = list( threshold = TrailingDistanceRange ),
> > label = "StopTrailingLONG")
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = "chain",
> > component.label = "StopLossShort",
> > variable = list( threshold = StopLossDistanceRange ),
> > label = "StopLossSHORT")
> >
> > add.distribution(strategy.st,
> > paramset.label = "MACD_OPT",
> > component.type = "chain",
> > component.label = "StopTrailingShort",
> > variable = list( threshold = TrailingDistanceRange ),
> > label = "StopTrailingSHORT")
> >
> > add.distribution.constraint(strategy.st,
> > paramset.label = "MACD_OPT",
> > distribution.label.1 = "StopLossLONG",
> > distribution.label.2 = "StopLossSHORT",
> > operator = "==",
> > label = "StoplossEquality")
> >
> > add.distribution.constraint(strategy.st,
> > paramset.label = "MACD_OPT",
> > distribution.label.1 = "StopTrailingLONG",
> > distribution.label.2 = "StopTrailingSHORT",
> > operator = "==",
> > label = "TrailingStopEquality")
> >
> > summary(getStrategy(strategy.st))
> >
> >
> >
> > start.t <- Sys.time()
> > paramsetenv<-new.env()
> > cl <- snow::makeCluster(no.cores, type = "SOCK")
> > registerDoSNOW(cl)
> > results <- apply.paramset(strategy.st,paramset.label="MACD_OPT",
> > portfolio=strategy.st, account=strategy.st,nsamples=0,verbose
> > = FALSE,
> > audit=paramsetenv)
> > snow::stopCluster(cl)
> > finish.t <- Sys.time()
> > print(finish.t-start.t)
> >
> >
> >
> >
> >
> > If there are any problems with the code when reproducing, sorry. I ran it
> > on my computer and it did seem to work as it did normally.
> >
> >
> >
> > 2016-04-06 21:05 GMT+03:00 Brian G. Peterson <brian at braverock.com>:
> >
> >> On Wed, 2016-04-06 at 20:58 +0300, Jersey Fanatic wrote:
> >> > Hi everyone,
> >> >
> >> > I am trying to backtest a simple strategy of mine, but it took approx 10
> >> > hours of processing so I was wondering if it is normal.
> >> >
> >> > My computer has an i7 core 2.10 GHz, 8GB RAM with Windows 7 Ultimate
> >> OS. My
> >> > code uses doSNOW package for parallel processing and all the 8 cores, so
> >> > the CPU is %100 all the time for the complete 10 hours.
> >> >
> >> > The data the backtest is done on is M5 OHLC FX (24 hour) data for
> >> approx 1
> >> > year which makes a total of 58000 data points (counting OHLC as 1). The
> >> > strategy has 4 entry&exit and 4 stoploss&trailingstoploss&takeprofit
> >> rules.
> >> > 144 combinations of parameters are tested.
> >> >
> >> > I would just like to know if this time is normal for just 1 home-use
> >> > computer. And I would be grateful for any recommendations to speed up
> >> this
> >> > process to decrease computing time.
> >>
> >> No, this sounds very high.
> >>
> >> You haven't told us enough about what you're trying to do for anyone
> >> here to help you in a specific way. You'd need to provide a minimal
> >> reproducible example.
> >>
> >> However, we typically use a benchmark time of one core minute per day
> >> per instrument on tick data (millions of observations per day).
> >>
> >> Is seems likely that you are doing too much path dependent looping,
> >> copying, or both.
> >>
> >> Regards,
> >>
> >> Brian
> >>
> >>
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-SIG-Finance at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>
_______________________________________________
R-SIG-Finance at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
-------------- next part --------------
Visit our website at http://www.ubs.com
This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.
E-mails are not encrypted and cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The sender
therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of e-mail transmission.
If verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities
or related financial instruments.
UBS Limited is a company limited by shares incorporated in the United
Kingdom registered in England and Wales with number 2035362.
Registered Office: 1 Finsbury Avenue, London EC2M 2PP
UBS Limited is authorised by the Prudential Regulation Authority
and regulated by the Financial Conduct Authority and the Prudential
Regulation Authority.
UBS AG is a public company incorporated with limited liability in
Switzerland domiciled in the Canton of Basel-City and the Canton of
Zurich respectively registered at the Commercial Registry offices in
those Cantons with new Identification No: CHE-101.329.561 as from 18
December 2013 (and prior to 18 December 2013 with Identification
No: CH-270.3.004.646-4) and having respective head offices at
Aeschenvorstadt 1, 4051 Basel and Bahnhofstrasse 45, 8001 Zurich,
Switzerland and is authorised and regulated by the Financial Market
Supervisory Authority in Switzerland. Registered in the United
Kingdom as a foreign company with No: FC021146 and having a UK
Establishment registered at Companies House, Cardiff, with
No: BR 004507. The principal office of UK Establishment: 1 Finsbury
Avenue, London EC2M 2PP. In the United Kingdom, UBS AG is authorised
by the Prudential Regulation Authority and subject to regulation
by the Financial Conduct Authority and limited regulation by the
Prudential Regulation Authority. Details about the extent of our
regulation by the Prudential Regulation Authority are available
from us on request.
UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.
More information about the R-SIG-Finance
mailing list