[R-SIG-Finance] checkBlotterUpdate fails on quantstrat
Erol Biceroglu
erol.biceroglu at alumni.utoronto.ca
Thu Jul 23 03:02:26 CEST 2015
Hello,
So I've taken your advice Joshua, and ran applyIndicators, applySignals and
applyRules one by one. What I've discovered is that my initial thoughts
that the NAs in the mktdata were being carried to updatePortf() were
incorrect, since I modified the signal and rules to ensure there were no
NAs, and still got the same error (checkBlotterUpdate fails).
Each function (applyIndicators, applySignals and applyRules) runs without
errors. I can even run applyStrategy and chart.Posn successfully.
What (I believe) is ultimately throwing it off is a duplicate entry in the
portfolio object (after running updatePortf()), where in the 2nd instance
there are NAs in Windows, or NaNs in Ubuntu, in:
-Pos.Value
-Period.Unrealized.PL
-Gross.Trading.PL
-Net.Trading.PL
I *think* this is the case since I can run perTradeStats() without any
issues.
Running the following after executing the code (provided below) will
demonstrate what I'm referring to:
getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
I've ran it in both Windows and Ubuntu OS's, under daily and weekly
frequencies.
In both OS's, daily frequencies cause checkBlotterUpdate to fail, whereas
under the weekly frequency, both OS's run checkBlotterUpdate successfully,
which allows me to generate tradeStats, and run additional functionality.
Here's my Windows sessionInfo():
> sessionInfo()
R version 3.2.0 (2015-04-16)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8 x64 (build 9200)
locale:
[1] LC_COLLATE=English_Canada.1252 LC_CTYPE=English_Canada.1252
LC_MONETARY=English_Canada.1252
[4] LC_NUMERIC=C LC_TIME=English_Canada.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quantstrat_0.9.1632 foreach_1.4.2
blotter_0.9.1637
[4] PerformanceAnalytics_1.4.3541 FinancialInstrument_1.2.0
quantmod_0.4-4
[7] TTR_0.22-0 xts_0.9-7 zoo_1.7-12
loaded via a namespace (and not attached):
[1] tools_3.2.0 codetools_0.2-11 grid_3.2.0 iterators_1.0.7
lattice_0.20-31
and here's my Ubuntu sessionInfo():
> sessionInfo()R version 3.2.1 (2015-06-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.2 LTS
locale:
[1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
[5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
LC_PAPER=en_CA.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quantstrat_0.9.1687 foreach_1.4.2
blotter_0.9.1666 PerformanceAnalytics_1.4.3541
[5] FinancialInstrument_1.2.0 quantmod_0.4-4
TTR_0.23-0 xts_0.9-7
[9] zoo_1.7-12
loaded via a namespace (and not attached):
[1] tools_3.2.1 codetools_0.2-8 grid_3.2.1 iterators_1.0.7
lattice_0.20-31
Lastly, here's the updated code, which I've attempted to reduce as much as
possible. Please let me know if there's any additional information that I
can provide.
Any thoughts and advice on how to proceed would be greatly appreciated.
Thank you for your help.
BEGIN CODE-----
library(quantstrat)
Sys.setenv(TZ="UTC")
startDate<-as.Date("1993-02-02", format="%Y-%m-%d")
endDate<-as.Date("2015-07-17", format="%Y-%m-%d")
getSymbols(Symbols = c("SPY", "^VIX")
,src="yahoo"
, verbose=TRUE
, warnings=TRUE
, auto.assign=TRUE
, return.class = "xts"
, index.class = "Date"
, from = startDate
, to = endDate
)
#set Financial instrument objects
currency("USD")
stock(primary_id = c("SPY", "VIX"),currency = "USD")
#name
tradeStrategy <-"SPYVIXStrategy"
#Date, one day before prices
strategyDate <- min(index(SPY)) - 1
#Code to reset
#rm.strat(tradeStrategy)
#rm(mktdata)
VIXThreshold <- 20
#RUNNING THIS WORKS
# VIX <- to.weekly(VIX, drop.time = FALSE)
# SPY <- to.weekly(SPY, drop.time = FALSE)
#init portfolio and account
initPortf(name = tradeStrategy
, symbols = "SPY" #as defined in Financial instrument
, initDate = strategyDate)
initAcct(name = tradeStrategy
,portfolios = tradeStrategy
,initDate = strategyDate
,initEq = 10e6 #as.vector(first(SPY$SPY.Close))*NumSh
)
#order book, and strategy
initOrders(portfolio = tradeStrategy
, initDate = strategyDate
)
#store strategy
strategy( tradeStrategy, store = TRUE)
#Pass VIX Close as an Indicator
PctStrat <- function(x){return(x)}
#add indicator
add.indicator(strategy = tradeStrategy
, name = "PctStrat"
, arguments = list( x = quote(VIX$VIX.Close))
, label = "VIXPct"
)
#>VIX 20
add.signal(strategy = tradeStrategy
, name = "sigThreshold"
, arguments = list(column = c("VIX.Close.VIXPct")
, threshold = quote(VIXThreshold) #20
, relationship = "gt"
, cross = TRUE #FALSE
)
, label = "HighVolatility"
)
#<=VIX 20
add.signal(strategy = tradeStrategy
, name = "sigThreshold"
, arguments = list(column = c("VIX.Close.VIXPct")
, threshold = quote(VIXThreshold) #20
, relationship = "lte"
, cross = TRUE #FALSE
)
, label = "LowVolatility"
)
#rules
add.rule(strategy = tradeStrategy
, name = "ruleSignal"
, arguments = list(sigcol="LowVolatility"
, sigval=TRUE
, orderqty=1
, ordertype="market"
, orderside=NULL#"long"
#, osFUN = "osMaxPos"
)
, type = "enter"
)
add.rule(strategy = tradeStrategy
, name = "ruleSignal"
, arguments = list(sigcol= "HighVolatility"
, sigval=TRUE
, orderqty="all"
, ordertype="market"
, orderside=NULL#"long"
#, osFUN = "osMaxPos"
)
, type = "exit"
)
applyStrategy(strategy = tradeStrategy
, portfolios = tradeStrategy
, debug = TRUE
)
#update
updatePortf(tradeStrategy)
updateAcct(tradeStrategy)
updateEndEq(tradeStrategy)
##Verify Results
###From Guy Yollin's Slides
checkBlotterUpdate <- function(port.st,account.st,verbose=TRUE)
{
ok <- TRUE
p <- getPortfolio(port.st)
a <- getAccount(account.st)
syms <- names(p$symbols)
port.tot <- sum(sapply(syms,FUN = function(x) eval(parse(
text=paste("sum(p$symbols",x,"posPL.USD$Net.Trading.PL)",sep="$")))))
port.sum.tot <- sum(p$summary$Net.Trading.PL)
if( !isTRUE(all.equal(port.tot,port.sum.tot)) ) {
ok <- FALSE
if( verbose )
print("portfolio P&L doesn't match sum of symbols P&L")
}
initEq <- as.numeric(first(a$summary$End.Eq))
endEq <- as.numeric(last(a$summary$End.Eq))
if( !isTRUE(all.equal(port.tot,endEq-initEq)) ) {
ok <- FALSE
if( verbose )
print("portfolio P&L doesn't match account P&L")
}
if( sum(duplicated(index(p$summary))) ) {
ok <- FALSE
if( verbose )
print("duplicate timestamps in portfolio summary")
}
if( sum(duplicated(index(a$summary))) ) {
ok <- FALSE
if( verbose )
print("duplicate timestamps in account summary")
}
return(ok)
}
###End Guy Yollin's code
#This fails
checkBlotterUpdate(tradeStrategy,tradeStrategy)
#Here's what I suspect is causing checkBlotterUpdate() to fail (see NA's in
Windows, NaN's on Ubuntu, line 2)
getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
#Chart strategy works
chart.Posn(tradeStrategy
, Symbol = "SPY"
#, Dates = "1994::"
#, Dates = "2012::"
,TA = "add_TA(VIX$VIX.Close)"
)
#this doesn't work
tradeStats(tradeStrategy)
#but this does
head(perTradeStats(tradeStrategy, "SPY"))
END CODE-------
Erol Biceroglu
*erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
On Tue, Jul 21, 2015 at 12:49 AM, Erol Biceroglu <
erol.biceroglu at alumni.utoronto.ca> wrote:
> Hi Joshua,
>
> Thanks for the feedback, your comments are helpful.
>
> My apologies for the confusion, perhaps I provided too much information at
> first. I am using apply.paramset, and although it runs, it doesn't work.
> It will finish without an error, but the results are empty. The NA is
> throwing off the aggregation of the trade statistics, which occurs before I
> get to apply.paramset, so it would have made my code much longer.
>
> My intention, and what I was hoping I did, was identify what as causing
> the error (the NAs in posPL):
>
> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>
> I tried to trim the example down, however I wasn't sure what was causing
> the error, so I also tried to preserve as much as I could. However I
> appreciate the advice, and I will keep that in mind in the future.
>
> Anyway, based on your comments, it looks like its the following rules that
> are causing the issue:
>
> Pct.lt.3qt.LowVol
>
> Pct.gte.3qt.LowVol
>
>
> These generate NA on the first time stamp, so I have something to go on to continue my investigation.
>
>
> Lastly, here's sessionInfo() output in case it's helpful.
>
>
> Thank you,
>
>
> > sessionInfo()R version 3.2.1 (2015-06-18)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 14.04.2 LTS
>
> locale:
> [1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
> [5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 LC_PAPER=en_CA.UTF-8 LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
>
> other attached packages:
> [1] IKTrading_1.0 roxygen2_4.1.1 digest_0.6.8 Rcpp_0.11.6
> [5] stringr_1.0.0 timeDate_3012.100 quantstrat_0.9.1687 foreach_1.4.2
> [9] blotter_0.9.1666 PerformanceAnalytics_1.4.3541 FinancialInstrument_1.2.0 quantmod_0.4-4
> [13] TTR_0.23-0 xts_0.9-7 zoo_1.7-12
>
> loaded via a namespace (and not attached):
> [1] lattice_0.20-31 codetools_0.2-8 grid_3.2.1 magrittr_1.5 stringi_0.5-5 iterators_1.0.7 tools_3.2.1
>
>
>
>
>
> Erol Biceroglu
>
>
> *erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
>
> On Mon, Jul 20, 2015 at 11:47 PM, Joshua Ulrich <josh.m.ulrich at gmail.com>
> wrote:
>
>> On Mon, Jul 20, 2015 at 7:53 PM, Erol Biceroglu
>> <erol.biceroglu at alumni.utoronto.ca> wrote:
>> > Hello,
>> >
>> > I've been playing around with quanstrat and was looking forward to
>> running
>> > *apply.paramset()* to optimize the strategy's parameters, only to find
>> an
>> > empty set of results.
>> >
>> Empty set of results from what? There isn't a call to apply.paramset
>> in your code. If you are actually running apply.paramset, please
>> provide the output from sessionInfo().
>>
>> > (Please note, my actual code follows after my message)
>> >
>> > After investigating, I found that *checkBlotterUpdate* fails with the
>> > message:
>> >
>> >> checkBlotterUpdate(tradeStrategy,tradeStrategy)
>> > [1] "portfolio P&L doesn't match sum of symbols P&L"
>> > [1] "portfolio P&L doesn't match account P&L"
>> > [1] FALSE
>> >
>> > Upon further investigation, I'm unable to run the following:
>> >> tradeStats(tradeStrategy)
>> > NULL
>> > Warning message:
>> > In tradeStats(tradeStrategy) : TotalNetProfit NA forSPY
>> >
>> > When I run this line (after executing everything), I find that the
>> > "strategy date" (the first date before the beginning of the time
>> series) is
>> > duplicated, and in the 2nd instance, there are NAs in:
>> > -Pos.Value
>> > -Period.Unrealized.PL
>> > -Gross.Trading.PL
>> > -Net.Trading.PL
>> >
>>
>> > I've tried to play around with it and unfortunately I can't figure out
>> what
>> > would cause the duplicate, generate the NA. Any thoughts or feedback
>> would
>> > be greatly appreciated. Thanks for your help.
>> >
>> You will likely get more help if you provide a _minimal_ reproducible
>> example (yours didn't run for me because I don't have IKTrading
>> installed). It would also help to provide more detail about what
>> you've tried in order to solve your problem--"tried to play around
>> with it" doesn't help people know what you did.
>>
>> The general advice I can give you is to individually run
>> applyIndicators, applySignals, and applyRules; and to check the
>> mktdata object after you run each function.
>>
>> > Here's the code:
>> > ---------
>> > rm(list=ls())
>> > library(quantstrat)
>> > library(timeDate)
>> > library(stringr)
>> > library(IKTrading)
>> >
>> >
>> > Sys.setenv(TZ="UTC")
>> > startDate<-as.Date("1993-02-02", format="%Y-%m-%d")
>> > endDate<-as.Date("2015-07-17", format="%Y-%m-%d")
>> >
>> > getSymbols(Symbols = c("SPY", "^VIX")
>> > ,src="yahoo"
>> > , verbose=TRUE
>> > , warnings=TRUE
>> > , auto.assign=TRUE
>> > , return.class = "xts"
>> > , index.class = "Date"
>> > ,from = startDate
>> > , to = endDate
>> > )
>> >
>> > #set Financial instrument objects
>> > currency("USD")
>> > stock(primary_id = c("SPY", "VIX"),currency = "USD")
>> >
>> >
>> > #name
>> > tradeStrategy <-"SPYVIXStrategy"
>> >
>> > #Date, one day before prices
>> > strategyDate <- min(index(SPY)) - 1
>> >
>> >
>> > #rm.strat(tradeStrategy)
>> > #rm(mktdata)
>> >
>> > NumSh<-3
>> > VIXThreshold <- 20
>> > PctThreshold <- 0.75
>> >
>> > #init portfolio and account
>> > initPortf(name = tradeStrategy
>> > , symbols = "SPY" #as defined in Financial instrument
>> > , initDate = strategyDate)
>> >
>> > initAcct(name = tradeStrategy
>> > ,portfolios = tradeStrategy
>> > ,initDate = strategyDate
>> > ,initEq = 10e6 #as.vector(first(SPY$SPY.Close))*NumSh
>> > )
>> >
>> >
>> >
>> > #order book, and strategy
>> > initOrders(portfolio = tradeStrategy
>> > , initDate = strategyDate
>> > )
>> >
>> > #position limits
>> > addPosLimit(tradeStrategy, symbol = "SPY", strategyDate, maxpos = NumSh,
>> > longlevels = NumSh)
>> >
>> > strategy( tradeStrategy, store = TRUE)
>> >
>> >
>> >
>> > #define indicator function
>> > PctStrat <- function(x){
>> > data<-merge(
>> > runPercentRank(x=ifelse(diff(x)<0,0,diff(x)),cumulative = FALSE
>> > )
>> > , ifelse(diff(x)<0,0,diff(x))
>> > , x
>> > )
>> > names(data) <- c("Percentile", "PositiveDiffs","VIX.Close")
>> >
>> > data<-xts(x = sapply(data
>> > ,function(x){ifelse(is.na(x),0,x)}
>> > ), order.by = index(data)
>>
>> > )
>> >
>> > return(data)
>> > }
>> >
>> >
>> > #add indicator
>> > add.indicator(strategy = tradeStrategy
>> > , name = "PctStrat"
>> > , arguments = list( x = quote(VIX$VIX.Close))
>> > , label = "VIXPct"
>> > )
>> >
>> >
>> > # >=75th percentile move
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigThreshold"
>> > , arguments = list(column = c("Percentile.VIXPct")
>> > , threshold = quote(PctThreshold) #0.75
>> > , relationship = "gte"
>> > , cross = FALSE
>> > )
>> > , label = "Pct.gte.3Qt"
>> > )
>> >
>> > #<75th percentile move
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigThreshold"
>> > , arguments = list(column = c("Percentile.VIXPct")
>> > , threshold = quote(PctThreshold) #0.75
>> > , relationship = "lt"
>> > , cross = FALSE
>> > )
>> > , label = "Pct.lt.3Qt"
>> > )
>> >
>> > #>VIX 20
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigThreshold"
>> > , arguments = list(column = c("VIX.Close.VIXPct")
>> > , threshold = quote(VIXThreshold) #20
>> > , relationship = "gt"
>> > , cross = FALSE
>> > )
>> > , label = "HighVolatility"
>> > )
>> >
>> >
>> > #<=VIX 20
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigThreshold"
>> > , arguments = list(column = c("VIX.Close.VIXPct")
>> > , threshold = quote(VIXThreshold) #20
>> > , relationship = "lte"
>> > , cross = FALSE
>> > )
>> > , label = "LowVolatility"
>> > )
>> >
>> > #intersect signals
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigAND"
>> > , arguments = list(columns = c("Pct.lt.3Qt",
>> "LowVolatility"),
>> > cross = TRUE)
>> > , label = "Pct.lt.3qt.LowVol"
>> > )
>> >
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigAND"
>> > , arguments = list(columns = c("Pct.lt.3Qt",
>> "HighVolatility"),
>> > cross = FALSE)
>> > , label = "Pct.lt.3qt.HighVol"
>> > )
>> >
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigAND"
>> > , arguments = list(columns = c("Pct.gte.3Qt",
>> "HighVolatility"),
>> > cross = FALSE)
>> > , label = "Pct.gte.3qt.HighVol"
>> > )
>> >
>> > add.signal(strategy = tradeStrategy
>> > , name = "sigAND"
>> > , arguments = list(columns = c("Pct.gte.3Qt",
>> "LowVolatility"),
>> > cross = TRUE)
>> > , label = "Pct.gte.3qt.LowVol"
>> > )
>> >
>> > #rules
>> > add.rule(strategy = tradeStrategy
>> > , name = "ruleSignal"
>> > , arguments = list(sigcol="Pct.lt.3qt.LowVol"
>> > , sigval=TRUE
>> > , orderqty=NumSh
>> > , ordertype="market"
>> > , orderside=NULL#"long"
>> > , osFUN = "osMaxPos"
>> > )
>> > , type = "enter"
>> > )
>> >
>> > add.rule(strategy = tradeStrategy
>> > , name = "ruleSignal"
>> > , arguments = list(sigcol= "HighVolatility"
>> #"Pct.lt.3qt.HighVol"
>> > , sigval=TRUE
>> > , orderqty="all"
>> > , ordertype="market"
>> > , orderside=NULL#"long"
>> > , osFUN = "osMaxPos"
>> > )
>> > , type = "exit"
>> > )
>> >
>> > add.rule(strategy = tradeStrategy
>> > , name = "ruleSignal"
>> > , arguments = list(sigcol="Pct.gte.3qt.LowVol"
>> > , sigval=TRUE
>> > , orderqty=1
>> > , ordertype="market"
>> > , orderside=NULL#"long"
>> > , osFUN = "osMaxPos"
>> > )
>> > , type = "exit"
>> > )
>> >
>> >
>> > applyStrategy(strategy = tradeStrategy
>> > , portfolios = tradeStrategy
>> > )
>> >
>> > updatePortf(tradeStrategy)
>> > updateAcct(tradeStrategy)
>> > updateEndEq(tradeStrategy)
>> >
>> >
>> > ###From Guy Yollin's Slides
>> > checkBlotterUpdate <- function(port.st,account.st,verbose=TRUE)
>> > {
>> > ok <- TRUE
>> > p <- getPortfolio(port.st)
>> > a <- getAccount(account.st)
>> > syms <- names(p$symbols)
>> > port.tot <- sum(sapply(syms,FUN = function(x) eval(parse(
>> > text=paste("sum(p$symbols",x,"posPL.USD$Net.Trading.PL
>> )",sep="$")))))
>> > port.sum.tot <- sum(p$summary$Net.Trading.PL)
>>
>> > if( !isTRUE(all.equal(port.tot,port.sum.tot)) ) {
>> > ok <- FALSE
>> > if( verbose )
>> > print("portfolio P&L doesn't match sum of symbols P&L")
>> > }
>> > initEq <- as.numeric(first(a$summary$End.Eq))
>> > endEq <- as.numeric(last(a$summary$End.Eq))
>> > if( !isTRUE(all.equal(port.tot,endEq-initEq)) ) {
>> > ok <- FALSE
>> > if( verbose )
>> > print("portfolio P&L doesn't match account P&L")
>> > }
>> > if( sum(duplicated(index(p$summary))) ) {
>> > ok <- FALSE
>> > if( verbose )
>> > print("duplicate timestamps in portfolio summary")
>> > }
>> > if( sum(duplicated(index(a$summary))) ) {
>> > ok <- FALSE
>> > if( verbose )
>> > print("duplicate timestamps in account summary")
>> > }
>> > return(ok)
>> > }
>> > ###End Guy Yollin's code
>> >
>> > #This fails
>> > checkBlotterUpdate(tradeStrategy,tradeStrategy)
>> >
>> > chart.Posn(tradeStrategy
>> > , Symbol = "SPY"
>> > #, Dates = "1994::"
>> > #, Dates = "2012::"
>> > ,TA = "add_TA(VIX$VIX.Close)"
>> >
>> > )
>> >
>> > #Here's an error
>> > tradeStats(tradeStrategy)
>> >
>> > #Here's the source of the error, an NA on the 2nd line
>> > getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>> >
>> > -------
>> > End code
>> >
>> >
>> > Erol Biceroglu
>> >
>> >
>> > *erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
>> >
>> > [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > R-SIG-Finance at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> > -- Subscriber-posting only. If you want to post, subscribe first.
>> > -- Also note that this is not the r-help list where general R questions
>> should go.
>>
>>
>>
>> --
>> Joshua Ulrich | about.me/joshuaulrich
>> FOSS Trading | www.fosstrading.com
>>
>
>
[[alternative HTML version deleted]]
More information about the R-SIG-Finance
mailing list