[R-SIG-Finance] checkBlotterUpdate fails on quantstrat
Joshua Ulrich
josh.m.ulrich at gmail.com
Fri Jul 24 06:00:21 CEST 2015
On Wed, Jul 22, 2015 at 8:02 PM, Erol Biceroglu
<erol.biceroglu at alumni.utoronto.ca> wrote:
> Hello,
>
> So I've taken your advice Joshua, and ran applyIndicators, applySignals and
> applyRules one by one. What I've discovered is that my initial thoughts
> that the NAs in the mktdata were being carried to updatePortf() were
> incorrect, since I modified the signal and rules to ensure there were no
> NAs, and still got the same error (checkBlotterUpdate fails).
>
> Each function (applyIndicators, applySignals and applyRules) runs without
> errors. I can even run applyStrategy and chart.Posn successfully.
>
> What (I believe) is ultimately throwing it off is a duplicate entry in the
> portfolio object (after running updatePortf()), where in the 2nd instance
> there are NAs in Windows, or NaNs in Ubuntu, in:
> -Pos.Value
> -Period.Unrealized.PL
> -Gross.Trading.PL
> -Net.Trading.PL
>
> I *think* this is the case since I can run perTradeStats() without any
> issues.
>
> Running the following after executing the code (provided below) will
> demonstrate what I'm referring to:
> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>
> I've ran it in both Windows and Ubuntu OS's, under daily and weekly
> frequencies.
>
> In both OS's, daily frequencies cause checkBlotterUpdate to fail, whereas
> under the weekly frequency, both OS's run checkBlotterUpdate successfully,
> which allows me to generate tradeStats, and run additional functionality.
>
> Here's my Windows sessionInfo():
<snip>
>
> and here's my Ubuntu sessionInfo():
<snip>
>
> Lastly, here's the updated code, which I've attempted to reduce as much as
> possible. Please let me know if there's any additional information that I
> can provide.
>
Thank you very much for the more minimal example. This looks like a
bug in blotter:::.updatePosPL. If you don't supply the Dates argument
in the updatePortf call, the dates are extracted from the index of the
Prices argument (or the object containing data for the given Symbol).
In this case, they're extracted from the SPY object, which has an Date
class index. The index of the posPL and txn tables are always
POSIXct.
In order to get a date range for which position P&L needs to be
updated, we subtract a very small value from the first observation in
Dates. Since Dates is a 'Date' vector in this case and subtracting a
very small value causes the date to shift back an entire day. This
causes the initializing transaction in the txn table to be included in
the P&L calculations. For example:
R> as.Date("1993-02-02")-0.0001
[1] "1993-02-01"
Here's a patch that seems to fix this specific issue. It needs more
testing before I'd be comfortable committing it to the repository.
Index: updatePosPL.R
===================================================================
--- updatePosPL.R (revision 1692)
+++ updatePosPL.R (working copy)
@@ -37,7 +37,7 @@
# if no date is specified, get all available dates
if(is.null(Dates)) {
- Dates = index(prices)
+ Dates = as.POSIXct(index(prices))
} else if(!is.timeBased(Dates)) {
Dates<- if(is.na(.parseISO8601(Dates)$first.time) ||
.parseISO8601(Dates)$first.time <
as.POSIXct(first(index(prices)))){
> Any thoughts and advice on how to proceed would be greatly appreciated.
>
> Thank you for your help.
>
> BEGIN CODE-----
<snip>
> END CODE-------
>
>
>
> Erol Biceroglu
> erol.biceroglu at alumni.utoronto.ca
>
>
> On Tue, Jul 21, 2015 at 12:49 AM, Erol Biceroglu
> <erol.biceroglu at alumni.utoronto.ca> wrote:
>>
>> Hi Joshua,
>>
>> Thanks for the feedback, your comments are helpful.
>>
>> My apologies for the confusion, perhaps I provided too much information at
>> first. I am using apply.paramset, and although it runs, it doesn't work.
>> It will finish without an error, but the results are empty. The NA is
>> throwing off the aggregation of the trade statistics, which occurs before I
>> get to apply.paramset, so it would have made my code much longer.
>>
>> My intention, and what I was hoping I did, was identify what as causing
>> the error (the NAs in posPL):
>>
>> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>>
>> I tried to trim the example down, however I wasn't sure what was causing
>> the error, so I also tried to preserve as much as I could. However I
>> appreciate the advice, and I will keep that in mind in the future.
>>
>> Anyway, based on your comments, it looks like its the following rules that
>> are causing the issue:
>>
>> Pct.lt.3qt.LowVol
>>
>> Pct.gte.3qt.LowVol
>>
>>
>> These generate NA on the first time stamp, so I have something to go on to
>> continue my investigation.
>>
>>
>> Lastly, here's sessionInfo() output in case it's helpful.
>>
>>
>> Thank you,
>>
>>
>> > sessionInfo()
>> R version 3.2.1 (2015-06-18)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>> Running under: Ubuntu 14.04.2 LTS
>>
>> locale:
>> [1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
>> LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
>> [5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>> LC_PAPER=en_CA.UTF-8 LC_NAME=C
>> [9] LC_ADDRESS=C LC_TELEPHONE=C
>> LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats graphics grDevices utils datasets methods base
>>
>> other attached packages:
>> [1] IKTrading_1.0 roxygen2_4.1.1
>> digest_0.6.8 Rcpp_0.11.6
>> [5] stringr_1.0.0 timeDate_3012.100
>> quantstrat_0.9.1687 foreach_1.4.2
>> [9] blotter_0.9.1666 PerformanceAnalytics_1.4.3541
>> FinancialInstrument_1.2.0 quantmod_0.4-4
>> [13] TTR_0.23-0 xts_0.9-7
>> zoo_1.7-12
>>
>> loaded via a namespace (and not attached):
>> [1] lattice_0.20-31 codetools_0.2-8 grid_3.2.1 magrittr_1.5
>> stringi_0.5-5 iterators_1.0.7 tools_3.2.1
>>
>>
>>
>>
>>
>> Erol Biceroglu
>> erol.biceroglu at alumni.utoronto.ca
>>
>>
>> On Mon, Jul 20, 2015 at 11:47 PM, Joshua Ulrich <josh.m.ulrich at gmail.com>
>> wrote:
>>>
>>> On Mon, Jul 20, 2015 at 7:53 PM, Erol Biceroglu
>>> <erol.biceroglu at alumni.utoronto.ca> wrote:
>>> > Hello,
>>> >
>>> > I've been playing around with quanstrat and was looking forward to
>>> > running
>>> > *apply.paramset()* to optimize the strategy's parameters, only to find
>>> > an
>>> > empty set of results.
>>> >
>>> Empty set of results from what? There isn't a call to apply.paramset
>>> in your code. If you are actually running apply.paramset, please
>>> provide the output from sessionInfo().
>>>
>>> > (Please note, my actual code follows after my message)
>>> >
>>> > After investigating, I found that *checkBlotterUpdate* fails with the
>>> > message:
>>> >
>>> >> checkBlotterUpdate(tradeStrategy,tradeStrategy)
>>> > [1] "portfolio P&L doesn't match sum of symbols P&L"
>>> > [1] "portfolio P&L doesn't match account P&L"
>>> > [1] FALSE
>>> >
>>> > Upon further investigation, I'm unable to run the following:
>>> >> tradeStats(tradeStrategy)
>>> > NULL
>>> > Warning message:
>>> > In tradeStats(tradeStrategy) : TotalNetProfit NA forSPY
>>> >
>>> > When I run this line (after executing everything), I find that the
>>> > "strategy date" (the first date before the beginning of the time
>>> > series) is
>>> > duplicated, and in the 2nd instance, there are NAs in:
>>> > -Pos.Value
>>> > -Period.Unrealized.PL
>>> > -Gross.Trading.PL
>>> > -Net.Trading.PL
>>> >
>>>
>>> > I've tried to play around with it and unfortunately I can't figure out
>>> > what
>>> > would cause the duplicate, generate the NA. Any thoughts or feedback
>>> > would
>>> > be greatly appreciated. Thanks for your help.
>>> >
>>> You will likely get more help if you provide a _minimal_ reproducible
>>> example (yours didn't run for me because I don't have IKTrading
>>> installed). It would also help to provide more detail about what
>>> you've tried in order to solve your problem--"tried to play around
>>> with it" doesn't help people know what you did.
>>>
>>> The general advice I can give you is to individually run
>>> applyIndicators, applySignals, and applyRules; and to check the
>>> mktdata object after you run each function.
>>>
>>> > Here's the code:
>>> > ---------
>>> > rm(list=ls())
>>> > library(quantstrat)
>>> > library(timeDate)
>>> > library(stringr)
>>> > library(IKTrading)
>>> >
>>> >
>>> > Sys.setenv(TZ="UTC")
>>> > startDate<-as.Date("1993-02-02", format="%Y-%m-%d")
>>> > endDate<-as.Date("2015-07-17", format="%Y-%m-%d")
>>> >
>>> > getSymbols(Symbols = c("SPY", "^VIX")
>>> > ,src="yahoo"
>>> > , verbose=TRUE
>>> > , warnings=TRUE
>>> > , auto.assign=TRUE
>>> > , return.class = "xts"
>>> > , index.class = "Date"
>>> > ,from = startDate
>>> > , to = endDate
>>> > )
>>> >
>>> > #set Financial instrument objects
>>> > currency("USD")
>>> > stock(primary_id = c("SPY", "VIX"),currency = "USD")
>>> >
>>> >
>>> > #name
>>> > tradeStrategy <-"SPYVIXStrategy"
>>> >
>>> > #Date, one day before prices
>>> > strategyDate <- min(index(SPY)) - 1
>>> >
>>> >
>>> > #rm.strat(tradeStrategy)
>>> > #rm(mktdata)
>>> >
>>> > NumSh<-3
>>> > VIXThreshold <- 20
>>> > PctThreshold <- 0.75
>>> >
>>> > #init portfolio and account
>>> > initPortf(name = tradeStrategy
>>> > , symbols = "SPY" #as defined in Financial instrument
>>> > , initDate = strategyDate)
>>> >
>>> > initAcct(name = tradeStrategy
>>> > ,portfolios = tradeStrategy
>>> > ,initDate = strategyDate
>>> > ,initEq = 10e6 #as.vector(first(SPY$SPY.Close))*NumSh
>>> > )
>>> >
>>> >
>>> >
>>> > #order book, and strategy
>>> > initOrders(portfolio = tradeStrategy
>>> > , initDate = strategyDate
>>> > )
>>> >
>>> > #position limits
>>> > addPosLimit(tradeStrategy, symbol = "SPY", strategyDate, maxpos =
>>> > NumSh,
>>> > longlevels = NumSh)
>>> >
>>> > strategy( tradeStrategy, store = TRUE)
>>> >
>>> >
>>> >
>>> > #define indicator function
>>> > PctStrat <- function(x){
>>> > data<-merge(
>>> > runPercentRank(x=ifelse(diff(x)<0,0,diff(x)),cumulative = FALSE
>>> > )
>>> > , ifelse(diff(x)<0,0,diff(x))
>>> > , x
>>> > )
>>> > names(data) <- c("Percentile", "PositiveDiffs","VIX.Close")
>>> >
>>> > data<-xts(x = sapply(data
>>> > ,function(x){ifelse(is.na(x),0,x)}
>>> > ), order.by = index(data)
>>>
>>> > )
>>> >
>>> > return(data)
>>> > }
>>> >
>>> >
>>> > #add indicator
>>> > add.indicator(strategy = tradeStrategy
>>> > , name = "PctStrat"
>>> > , arguments = list( x = quote(VIX$VIX.Close))
>>> > , label = "VIXPct"
>>> > )
>>> >
>>> >
>>> > # >=75th percentile move
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigThreshold"
>>> > , arguments = list(column = c("Percentile.VIXPct")
>>> > , threshold = quote(PctThreshold) #0.75
>>> > , relationship = "gte"
>>> > , cross = FALSE
>>> > )
>>> > , label = "Pct.gte.3Qt"
>>> > )
>>> >
>>> > #<75th percentile move
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigThreshold"
>>> > , arguments = list(column = c("Percentile.VIXPct")
>>> > , threshold = quote(PctThreshold) #0.75
>>> > , relationship = "lt"
>>> > , cross = FALSE
>>> > )
>>> > , label = "Pct.lt.3Qt"
>>> > )
>>> >
>>> > #>VIX 20
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigThreshold"
>>> > , arguments = list(column = c("VIX.Close.VIXPct")
>>> > , threshold = quote(VIXThreshold) #20
>>> > , relationship = "gt"
>>> > , cross = FALSE
>>> > )
>>> > , label = "HighVolatility"
>>> > )
>>> >
>>> >
>>> > #<=VIX 20
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigThreshold"
>>> > , arguments = list(column = c("VIX.Close.VIXPct")
>>> > , threshold = quote(VIXThreshold) #20
>>> > , relationship = "lte"
>>> > , cross = FALSE
>>> > )
>>> > , label = "LowVolatility"
>>> > )
>>> >
>>> > #intersect signals
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigAND"
>>> > , arguments = list(columns = c("Pct.lt.3Qt",
>>> > "LowVolatility"),
>>> > cross = TRUE)
>>> > , label = "Pct.lt.3qt.LowVol"
>>> > )
>>> >
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigAND"
>>> > , arguments = list(columns = c("Pct.lt.3Qt",
>>> > "HighVolatility"),
>>> > cross = FALSE)
>>> > , label = "Pct.lt.3qt.HighVol"
>>> > )
>>> >
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigAND"
>>> > , arguments = list(columns = c("Pct.gte.3Qt",
>>> > "HighVolatility"),
>>> > cross = FALSE)
>>> > , label = "Pct.gte.3qt.HighVol"
>>> > )
>>> >
>>> > add.signal(strategy = tradeStrategy
>>> > , name = "sigAND"
>>> > , arguments = list(columns = c("Pct.gte.3Qt",
>>> > "LowVolatility"),
>>> > cross = TRUE)
>>> > , label = "Pct.gte.3qt.LowVol"
>>> > )
>>> >
>>> > #rules
>>> > add.rule(strategy = tradeStrategy
>>> > , name = "ruleSignal"
>>> > , arguments = list(sigcol="Pct.lt.3qt.LowVol"
>>> > , sigval=TRUE
>>> > , orderqty=NumSh
>>> > , ordertype="market"
>>> > , orderside=NULL#"long"
>>> > , osFUN = "osMaxPos"
>>> > )
>>> > , type = "enter"
>>> > )
>>> >
>>> > add.rule(strategy = tradeStrategy
>>> > , name = "ruleSignal"
>>> > , arguments = list(sigcol= "HighVolatility"
>>> > #"Pct.lt.3qt.HighVol"
>>> > , sigval=TRUE
>>> > , orderqty="all"
>>> > , ordertype="market"
>>> > , orderside=NULL#"long"
>>> > , osFUN = "osMaxPos"
>>> > )
>>> > , type = "exit"
>>> > )
>>> >
>>> > add.rule(strategy = tradeStrategy
>>> > , name = "ruleSignal"
>>> > , arguments = list(sigcol="Pct.gte.3qt.LowVol"
>>> > , sigval=TRUE
>>> > , orderqty=1
>>> > , ordertype="market"
>>> > , orderside=NULL#"long"
>>> > , osFUN = "osMaxPos"
>>> > )
>>> > , type = "exit"
>>> > )
>>> >
>>> >
>>> > applyStrategy(strategy = tradeStrategy
>>> > , portfolios = tradeStrategy
>>> > )
>>> >
>>> > updatePortf(tradeStrategy)
>>> > updateAcct(tradeStrategy)
>>> > updateEndEq(tradeStrategy)
>>> >
>>> >
>>> > ###From Guy Yollin's Slides
>>> > checkBlotterUpdate <- function(port.st,account.st,verbose=TRUE)
>>> > {
>>> > ok <- TRUE
>>> > p <- getPortfolio(port.st)
>>> > a <- getAccount(account.st)
>>> > syms <- names(p$symbols)
>>> > port.tot <- sum(sapply(syms,FUN = function(x) eval(parse(
>>> >
>>> > text=paste("sum(p$symbols",x,"posPL.USD$Net.Trading.PL)",sep="$")))))
>>> > port.sum.tot <- sum(p$summary$Net.Trading.PL)
>>>
>>> > if( !isTRUE(all.equal(port.tot,port.sum.tot)) ) {
>>> > ok <- FALSE
>>> > if( verbose )
>>> > print("portfolio P&L doesn't match sum of symbols P&L")
>>> > }
>>> > initEq <- as.numeric(first(a$summary$End.Eq))
>>> > endEq <- as.numeric(last(a$summary$End.Eq))
>>> > if( !isTRUE(all.equal(port.tot,endEq-initEq)) ) {
>>> > ok <- FALSE
>>> > if( verbose )
>>> > print("portfolio P&L doesn't match account P&L")
>>> > }
>>> > if( sum(duplicated(index(p$summary))) ) {
>>> > ok <- FALSE
>>> > if( verbose )
>>> > print("duplicate timestamps in portfolio summary")
>>> > }
>>> > if( sum(duplicated(index(a$summary))) ) {
>>> > ok <- FALSE
>>> > if( verbose )
>>> > print("duplicate timestamps in account summary")
>>> > }
>>> > return(ok)
>>> > }
>>> > ###End Guy Yollin's code
>>> >
>>> > #This fails
>>> > checkBlotterUpdate(tradeStrategy,tradeStrategy)
>>> >
>>> > chart.Posn(tradeStrategy
>>> > , Symbol = "SPY"
>>> > #, Dates = "1994::"
>>> > #, Dates = "2012::"
>>> > ,TA = "add_TA(VIX$VIX.Close)"
>>> >
>>> > )
>>> >
>>> > #Here's an error
>>> > tradeStats(tradeStrategy)
>>> >
>>> > #Here's the source of the error, an NA on the 2nd line
>>> > getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>>> >
>>> > -------
>>> > End code
>>> >
>>> >
>>> > Erol Biceroglu
>>> >
>>> >
>>> > *erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
>>> >
>>> > [[alternative HTML version deleted]]
>>> >
>>> > _______________________________________________
>>> > R-SIG-Finance at r-project.org mailing list
>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>> > -- Subscriber-posting only. If you want to post, subscribe first.
>>> > -- Also note that this is not the r-help list where general R questions
>>> > should go.
>>>
>>>
>>>
>>> --
>>> Joshua Ulrich | about.me/joshuaulrich
>>> FOSS Trading | www.fosstrading.com
>>
>>
>
--
Joshua Ulrich | about.me/joshuaulrich
FOSS Trading | www.fosstrading.com
More information about the R-SIG-Finance
mailing list