[R-SIG-Finance] checkBlotterUpdate fails on quantstrat
Erol Biceroglu
erol.biceroglu at alumni.utoronto.ca
Sat Jul 25 03:08:59 CEST 2015
Hello Joshua,
I can also confirm that Brian's suggestion solves the issue (in both OSs).
Thank you both very much for your help and time with this, I very much
appreciate it.
Regards,
Erol Biceroglu
*erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
On Fri, Jul 24, 2015 at 9:41 AM, Joshua Ulrich <josh.m.ulrich at gmail.com>
wrote:
> On Thu, Jul 23, 2015 at 11:00 PM, Joshua Ulrich <josh.m.ulrich at gmail.com>
> wrote:
> > On Wed, Jul 22, 2015 at 8:02 PM, Erol Biceroglu
> > <erol.biceroglu at alumni.utoronto.ca> wrote:
> >> Hello,
> >>
> >> So I've taken your advice Joshua, and ran applyIndicators, applySignals
> and
> >> applyRules one by one. What I've discovered is that my initial thoughts
> >> that the NAs in the mktdata were being carried to updatePortf() were
> >> incorrect, since I modified the signal and rules to ensure there were no
> >> NAs, and still got the same error (checkBlotterUpdate fails).
> >>
> >> Each function (applyIndicators, applySignals and applyRules) runs
> without
> >> errors. I can even run applyStrategy and chart.Posn successfully.
> >>
> >> What (I believe) is ultimately throwing it off is a duplicate entry in
> the
> >> portfolio object (after running updatePortf()), where in the 2nd
> instance
> >> there are NAs in Windows, or NaNs in Ubuntu, in:
> >> -Pos.Value
> >> -Period.Unrealized.PL
> >> -Gross.Trading.PL
> >> -Net.Trading.PL
> >>
> >> I *think* this is the case since I can run perTradeStats() without any
> >> issues.
> >>
> >> Running the following after executing the code (provided below) will
> >> demonstrate what I'm referring to:
> >> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
> >>
> >> I've ran it in both Windows and Ubuntu OS's, under daily and weekly
> >> frequencies.
> >>
> >> In both OS's, daily frequencies cause checkBlotterUpdate to fail,
> whereas
> >> under the weekly frequency, both OS's run checkBlotterUpdate
> successfully,
> >> which allows me to generate tradeStats, and run additional
> functionality.
> >>
> >> Here's my Windows sessionInfo():
> > <snip>
> >>
> >> and here's my Ubuntu sessionInfo():
> > <snip>
> >>
> >> Lastly, here's the updated code, which I've attempted to reduce as much
> as
> >> possible. Please let me know if there's any additional information
> that I
> >> can provide.
> >>
> > Thank you very much for the more minimal example. This looks like a
> > bug in blotter:::.updatePosPL. If you don't supply the Dates argument
> > in the updatePortf call, the dates are extracted from the index of the
> > Prices argument (or the object containing data for the given Symbol).
> > In this case, they're extracted from the SPY object, which has an Date
> > class index. The index of the posPL and txn tables are always
> > POSIXct.
> >
> > In order to get a date range for which position P&L needs to be
> > updated, we subtract a very small value from the first observation in
> > Dates. Since Dates is a 'Date' vector in this case and subtracting a
> > very small value causes the date to shift back an entire day. This
> > causes the initializing transaction in the txn table to be included in
> > the P&L calculations. For example:
> > R> as.Date("1993-02-02")-0.0001
> > [1] "1993-02-01"
> >
> > Here's a patch that seems to fix this specific issue. It needs more
> > testing before I'd be comfortable committing it to the repository.
> >
> Brian Peterson suggested an easier/simpler fix: simply do not set the
> initDate in your calls to initPortf and initAcct. I've confirmed that
> avoids this issue.
>
> > Index: updatePosPL.R
> > ===================================================================
> > --- updatePosPL.R (revision 1692)
> > +++ updatePosPL.R (working copy)
> > @@ -37,7 +37,7 @@
> >
> > # if no date is specified, get all available dates
> > if(is.null(Dates)) {
> > - Dates = index(prices)
> > + Dates = as.POSIXct(index(prices))
> > } else if(!is.timeBased(Dates)) {
> > Dates<- if(is.na(.parseISO8601(Dates)$first.time) ||
> > .parseISO8601(Dates)$first.time <
> > as.POSIXct(first(index(prices)))){
> >
> >
> >> Any thoughts and advice on how to proceed would be greatly appreciated.
> >>
> >> Thank you for your help.
> >>
> >> BEGIN CODE-----
> > <snip>
> >> END CODE-------
> >>
> >>
> >>
> >> Erol Biceroglu
> >> erol.biceroglu at alumni.utoronto.ca
> >>
> >>
> >> On Tue, Jul 21, 2015 at 12:49 AM, Erol Biceroglu
> >> <erol.biceroglu at alumni.utoronto.ca> wrote:
> >>>
> >>> Hi Joshua,
> >>>
> >>> Thanks for the feedback, your comments are helpful.
> >>>
> >>> My apologies for the confusion, perhaps I provided too much
> information at
> >>> first. I am using apply.paramset, and although it runs, it doesn't
> work.
> >>> It will finish without an error, but the results are empty. The NA is
> >>> throwing off the aggregation of the trade statistics, which occurs
> before I
> >>> get to apply.paramset, so it would have made my code much longer.
> >>>
> >>> My intention, and what I was hoping I did, was identify what as causing
> >>> the error (the NAs in posPL):
> >>>
> >>> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
> >>>
> >>> I tried to trim the example down, however I wasn't sure what was
> causing
> >>> the error, so I also tried to preserve as much as I could. However I
> >>> appreciate the advice, and I will keep that in mind in the future.
> >>>
> >>> Anyway, based on your comments, it looks like its the following rules
> that
> >>> are causing the issue:
> >>>
> >>> Pct.lt.3qt.LowVol
> >>>
> >>> Pct.gte.3qt.LowVol
> >>>
> >>>
> >>> These generate NA on the first time stamp, so I have something to go
> on to
> >>> continue my investigation.
> >>>
> >>>
> >>> Lastly, here's sessionInfo() output in case it's helpful.
> >>>
> >>>
> >>> Thank you,
> >>>
> >>>
> >>> > sessionInfo()
> >>> R version 3.2.1 (2015-06-18)
> >>> Platform: x86_64-pc-linux-gnu (64-bit)
> >>> Running under: Ubuntu 14.04.2 LTS
> >>>
> >>> locale:
> >>> [1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
> >>> LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
> >>> [5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
> >>> LC_PAPER=en_CA.UTF-8 LC_NAME=C
> >>> [9] LC_ADDRESS=C LC_TELEPHONE=C
> >>> LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
> >>>
> >>> attached base packages:
> >>> [1] stats graphics grDevices utils datasets methods base
> >>>
> >>> other attached packages:
> >>> [1] IKTrading_1.0 roxygen2_4.1.1
> >>> digest_0.6.8 Rcpp_0.11.6
> >>> [5] stringr_1.0.0 timeDate_3012.100
> >>> quantstrat_0.9.1687 foreach_1.4.2
> >>> [9] blotter_0.9.1666 PerformanceAnalytics_1.4.3541
> >>> FinancialInstrument_1.2.0 quantmod_0.4-4
> >>> [13] TTR_0.23-0 xts_0.9-7
> >>> zoo_1.7-12
> >>>
> >>> loaded via a namespace (and not attached):
> >>> [1] lattice_0.20-31 codetools_0.2-8 grid_3.2.1 magrittr_1.5
> >>> stringi_0.5-5 iterators_1.0.7 tools_3.2.1
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Erol Biceroglu
> >>> erol.biceroglu at alumni.utoronto.ca
> >>>
> >>>
> >>> On Mon, Jul 20, 2015 at 11:47 PM, Joshua Ulrich <
> josh.m.ulrich at gmail.com>
> >>> wrote:
> >>>>
> >>>> On Mon, Jul 20, 2015 at 7:53 PM, Erol Biceroglu
> >>>> <erol.biceroglu at alumni.utoronto.ca> wrote:
> >>>> > Hello,
> >>>> >
> >>>> > I've been playing around with quanstrat and was looking forward to
> >>>> > running
> >>>> > *apply.paramset()* to optimize the strategy's parameters, only to
> find
> >>>> > an
> >>>> > empty set of results.
> >>>> >
> >>>> Empty set of results from what? There isn't a call to apply.paramset
> >>>> in your code. If you are actually running apply.paramset, please
> >>>> provide the output from sessionInfo().
> >>>>
> >>>> > (Please note, my actual code follows after my message)
> >>>> >
> >>>> > After investigating, I found that *checkBlotterUpdate* fails with
> the
> >>>> > message:
> >>>> >
> >>>> >> checkBlotterUpdate(tradeStrategy,tradeStrategy)
> >>>> > [1] "portfolio P&L doesn't match sum of symbols P&L"
> >>>> > [1] "portfolio P&L doesn't match account P&L"
> >>>> > [1] FALSE
> >>>> >
> >>>> > Upon further investigation, I'm unable to run the following:
> >>>> >> tradeStats(tradeStrategy)
> >>>> > NULL
> >>>> > Warning message:
> >>>> > In tradeStats(tradeStrategy) : TotalNetProfit NA forSPY
> >>>> >
> >>>> > When I run this line (after executing everything), I find that the
> >>>> > "strategy date" (the first date before the beginning of the time
> >>>> > series) is
> >>>> > duplicated, and in the 2nd instance, there are NAs in:
> >>>> > -Pos.Value
> >>>> > -Period.Unrealized.PL
> >>>> > -Gross.Trading.PL
> >>>> > -Net.Trading.PL
> >>>> >
> >>>>
> >>>> > I've tried to play around with it and unfortunately I can't figure
> out
> >>>> > what
> >>>> > would cause the duplicate, generate the NA. Any thoughts or
> feedback
> >>>> > would
> >>>> > be greatly appreciated. Thanks for your help.
> >>>> >
> >>>> You will likely get more help if you provide a _minimal_ reproducible
> >>>> example (yours didn't run for me because I don't have IKTrading
> >>>> installed). It would also help to provide more detail about what
> >>>> you've tried in order to solve your problem--"tried to play around
> >>>> with it" doesn't help people know what you did.
> >>>>
> >>>> The general advice I can give you is to individually run
> >>>> applyIndicators, applySignals, and applyRules; and to check the
> >>>> mktdata object after you run each function.
> >>>>
> >>>> > Here's the code:
> >>>> > ---------
> >>>> > rm(list=ls())
> >>>> > library(quantstrat)
> >>>> > library(timeDate)
> >>>> > library(stringr)
> >>>> > library(IKTrading)
> >>>> >
> >>>> >
> >>>> > Sys.setenv(TZ="UTC")
> >>>> > startDate<-as.Date("1993-02-02", format="%Y-%m-%d")
> >>>> > endDate<-as.Date("2015-07-17", format="%Y-%m-%d")
> >>>> >
> >>>> > getSymbols(Symbols = c("SPY", "^VIX")
> >>>> > ,src="yahoo"
> >>>> > , verbose=TRUE
> >>>> > , warnings=TRUE
> >>>> > , auto.assign=TRUE
> >>>> > , return.class = "xts"
> >>>> > , index.class = "Date"
> >>>> > ,from = startDate
> >>>> > , to = endDate
> >>>> > )
> >>>> >
> >>>> > #set Financial instrument objects
> >>>> > currency("USD")
> >>>> > stock(primary_id = c("SPY", "VIX"),currency = "USD")
> >>>> >
> >>>> >
> >>>> > #name
> >>>> > tradeStrategy <-"SPYVIXStrategy"
> >>>> >
> >>>> > #Date, one day before prices
> >>>> > strategyDate <- min(index(SPY)) - 1
> >>>> >
> >>>> >
> >>>> > #rm.strat(tradeStrategy)
> >>>> > #rm(mktdata)
> >>>> >
> >>>> > NumSh<-3
> >>>> > VIXThreshold <- 20
> >>>> > PctThreshold <- 0.75
> >>>> >
> >>>> > #init portfolio and account
> >>>> > initPortf(name = tradeStrategy
> >>>> > , symbols = "SPY" #as defined in Financial instrument
> >>>> > , initDate = strategyDate)
> >>>> >
> >>>> > initAcct(name = tradeStrategy
> >>>> > ,portfolios = tradeStrategy
> >>>> > ,initDate = strategyDate
> >>>> > ,initEq = 10e6 #as.vector(first(SPY$SPY.Close))*NumSh
> >>>> > )
> >>>> >
> >>>> >
> >>>> >
> >>>> > #order book, and strategy
> >>>> > initOrders(portfolio = tradeStrategy
> >>>> > , initDate = strategyDate
> >>>> > )
> >>>> >
> >>>> > #position limits
> >>>> > addPosLimit(tradeStrategy, symbol = "SPY", strategyDate, maxpos =
> >>>> > NumSh,
> >>>> > longlevels = NumSh)
> >>>> >
> >>>> > strategy( tradeStrategy, store = TRUE)
> >>>> >
> >>>> >
> >>>> >
> >>>> > #define indicator function
> >>>> > PctStrat <- function(x){
> >>>> > data<-merge(
> >>>> > runPercentRank(x=ifelse(diff(x)<0,0,diff(x)),cumulative = FALSE
> >>>> > )
> >>>> > , ifelse(diff(x)<0,0,diff(x))
> >>>> > , x
> >>>> > )
> >>>> > names(data) <- c("Percentile", "PositiveDiffs","VIX.Close")
> >>>> >
> >>>> > data<-xts(x = sapply(data
> >>>> > ,function(x){ifelse(is.na(x),0,x)}
> >>>> > ), order.by = index(data)
> >>>>
> >>>> > )
> >>>> >
> >>>> > return(data)
> >>>> > }
> >>>> >
> >>>> >
> >>>> > #add indicator
> >>>> > add.indicator(strategy = tradeStrategy
> >>>> > , name = "PctStrat"
> >>>> > , arguments = list( x = quote(VIX$VIX.Close))
> >>>> > , label = "VIXPct"
> >>>> > )
> >>>> >
> >>>> >
> >>>> > # >=75th percentile move
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigThreshold"
> >>>> > , arguments = list(column = c("Percentile.VIXPct")
> >>>> > , threshold = quote(PctThreshold)
> #0.75
> >>>> > , relationship = "gte"
> >>>> > , cross = FALSE
> >>>> > )
> >>>> > , label = "Pct.gte.3Qt"
> >>>> > )
> >>>> >
> >>>> > #<75th percentile move
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigThreshold"
> >>>> > , arguments = list(column = c("Percentile.VIXPct")
> >>>> > , threshold = quote(PctThreshold)
> #0.75
> >>>> > , relationship = "lt"
> >>>> > , cross = FALSE
> >>>> > )
> >>>> > , label = "Pct.lt.3Qt"
> >>>> > )
> >>>> >
> >>>> > #>VIX 20
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigThreshold"
> >>>> > , arguments = list(column = c("VIX.Close.VIXPct")
> >>>> > , threshold = quote(VIXThreshold) #20
> >>>> > , relationship = "gt"
> >>>> > , cross = FALSE
> >>>> > )
> >>>> > , label = "HighVolatility"
> >>>> > )
> >>>> >
> >>>> >
> >>>> > #<=VIX 20
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigThreshold"
> >>>> > , arguments = list(column = c("VIX.Close.VIXPct")
> >>>> > , threshold = quote(VIXThreshold) #20
> >>>> > , relationship = "lte"
> >>>> > , cross = FALSE
> >>>> > )
> >>>> > , label = "LowVolatility"
> >>>> > )
> >>>> >
> >>>> > #intersect signals
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigAND"
> >>>> > , arguments = list(columns = c("Pct.lt.3Qt",
> >>>> > "LowVolatility"),
> >>>> > cross = TRUE)
> >>>> > , label = "Pct.lt.3qt.LowVol"
> >>>> > )
> >>>> >
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigAND"
> >>>> > , arguments = list(columns = c("Pct.lt.3Qt",
> >>>> > "HighVolatility"),
> >>>> > cross = FALSE)
> >>>> > , label = "Pct.lt.3qt.HighVol"
> >>>> > )
> >>>> >
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigAND"
> >>>> > , arguments = list(columns = c("Pct.gte.3Qt",
> >>>> > "HighVolatility"),
> >>>> > cross = FALSE)
> >>>> > , label = "Pct.gte.3qt.HighVol"
> >>>> > )
> >>>> >
> >>>> > add.signal(strategy = tradeStrategy
> >>>> > , name = "sigAND"
> >>>> > , arguments = list(columns = c("Pct.gte.3Qt",
> >>>> > "LowVolatility"),
> >>>> > cross = TRUE)
> >>>> > , label = "Pct.gte.3qt.LowVol"
> >>>> > )
> >>>> >
> >>>> > #rules
> >>>> > add.rule(strategy = tradeStrategy
> >>>> > , name = "ruleSignal"
> >>>> > , arguments = list(sigcol="Pct.lt.3qt.LowVol"
> >>>> > , sigval=TRUE
> >>>> > , orderqty=NumSh
> >>>> > , ordertype="market"
> >>>> > , orderside=NULL#"long"
> >>>> > , osFUN = "osMaxPos"
> >>>> > )
> >>>> > , type = "enter"
> >>>> > )
> >>>> >
> >>>> > add.rule(strategy = tradeStrategy
> >>>> > , name = "ruleSignal"
> >>>> > , arguments = list(sigcol= "HighVolatility"
> >>>> > #"Pct.lt.3qt.HighVol"
> >>>> > , sigval=TRUE
> >>>> > , orderqty="all"
> >>>> > , ordertype="market"
> >>>> > , orderside=NULL#"long"
> >>>> > , osFUN = "osMaxPos"
> >>>> > )
> >>>> > , type = "exit"
> >>>> > )
> >>>> >
> >>>> > add.rule(strategy = tradeStrategy
> >>>> > , name = "ruleSignal"
> >>>> > , arguments = list(sigcol="Pct.gte.3qt.LowVol"
> >>>> > , sigval=TRUE
> >>>> > , orderqty=1
> >>>> > , ordertype="market"
> >>>> > , orderside=NULL#"long"
> >>>> > , osFUN = "osMaxPos"
> >>>> > )
> >>>> > , type = "exit"
> >>>> > )
> >>>> >
> >>>> >
> >>>> > applyStrategy(strategy = tradeStrategy
> >>>> > , portfolios = tradeStrategy
> >>>> > )
> >>>> >
> >>>> > updatePortf(tradeStrategy)
> >>>> > updateAcct(tradeStrategy)
> >>>> > updateEndEq(tradeStrategy)
> >>>> >
> >>>> >
> >>>> > ###From Guy Yollin's Slides
> >>>> > checkBlotterUpdate <- function(port.st,account.st,verbose=TRUE)
> >>>> > {
> >>>> > ok <- TRUE
> >>>> > p <- getPortfolio(port.st)
> >>>> > a <- getAccount(account.st)
> >>>> > syms <- names(p$symbols)
> >>>> > port.tot <- sum(sapply(syms,FUN = function(x) eval(parse(
> >>>> >
> >>>> > text=paste("sum(p$symbols",x,"posPL.USD$Net.Trading.PL
> )",sep="$")))))
> >>>> > port.sum.tot <- sum(p$summary$Net.Trading.PL)
> >>>>
> >>>> > if( !isTRUE(all.equal(port.tot,port.sum.tot)) ) {
> >>>> > ok <- FALSE
> >>>> > if( verbose )
> >>>> > print("portfolio P&L doesn't match sum of symbols P&L")
> >>>> > }
> >>>> > initEq <- as.numeric(first(a$summary$End.Eq))
> >>>> > endEq <- as.numeric(last(a$summary$End.Eq))
> >>>> > if( !isTRUE(all.equal(port.tot,endEq-initEq)) ) {
> >>>> > ok <- FALSE
> >>>> > if( verbose )
> >>>> > print("portfolio P&L doesn't match account P&L")
> >>>> > }
> >>>> > if( sum(duplicated(index(p$summary))) ) {
> >>>> > ok <- FALSE
> >>>> > if( verbose )
> >>>> > print("duplicate timestamps in portfolio summary")
> >>>> > }
> >>>> > if( sum(duplicated(index(a$summary))) ) {
> >>>> > ok <- FALSE
> >>>> > if( verbose )
> >>>> > print("duplicate timestamps in account summary")
> >>>> > }
> >>>> > return(ok)
> >>>> > }
> >>>> > ###End Guy Yollin's code
> >>>> >
> >>>> > #This fails
> >>>> > checkBlotterUpdate(tradeStrategy,tradeStrategy)
> >>>> >
> >>>> > chart.Posn(tradeStrategy
> >>>> > , Symbol = "SPY"
> >>>> > #, Dates = "1994::"
> >>>> > #, Dates = "2012::"
> >>>> > ,TA = "add_TA(VIX$VIX.Close)"
> >>>> >
> >>>> > )
> >>>> >
> >>>> > #Here's an error
> >>>> > tradeStats(tradeStrategy)
> >>>> >
> >>>> > #Here's the source of the error, an NA on the 2nd line
> >>>> > getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
> >>>> >
> >>>> > -------
> >>>> > End code
> >>>> >
> >>>> >
> >>>> > Erol Biceroglu
> >>>> >
> >>>> >
> >>>> > *erol.biceroglu at alumni.utoronto.ca <
> erol.biceroglu at alumni.utoronto.ca>*
> >>>> >
> >>>> > [[alternative HTML version deleted]]
> >>>> >
> >>>> > _______________________________________________
> >>>> > R-SIG-Finance at r-project.org mailing list
> >>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >>>> > -- Subscriber-posting only. If you want to post, subscribe first.
> >>>> > -- Also note that this is not the r-help list where general R
> questions
> >>>> > should go.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Joshua Ulrich | about.me/joshuaulrich
> >>>> FOSS Trading | www.fosstrading.com
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > Joshua Ulrich | about.me/joshuaulrich
> > FOSS Trading | www.fosstrading.com
>
>
>
> --
> Joshua Ulrich | about.me/joshuaulrich
> FOSS Trading | www.fosstrading.com
>
[[alternative HTML version deleted]]
More information about the R-SIG-Finance
mailing list