[R-SIG-Finance] checkBlotterUpdate fails on quantstrat
Joshua Ulrich
josh.m.ulrich at gmail.com
Fri Jul 24 15:41:23 CEST 2015
On Thu, Jul 23, 2015 at 11:00 PM, Joshua Ulrich <josh.m.ulrich at gmail.com> wrote:
> On Wed, Jul 22, 2015 at 8:02 PM, Erol Biceroglu
> <erol.biceroglu at alumni.utoronto.ca> wrote:
>> Hello,
>>
>> So I've taken your advice Joshua, and ran applyIndicators, applySignals and
>> applyRules one by one. What I've discovered is that my initial thoughts
>> that the NAs in the mktdata were being carried to updatePortf() were
>> incorrect, since I modified the signal and rules to ensure there were no
>> NAs, and still got the same error (checkBlotterUpdate fails).
>>
>> Each function (applyIndicators, applySignals and applyRules) runs without
>> errors. I can even run applyStrategy and chart.Posn successfully.
>>
>> What (I believe) is ultimately throwing it off is a duplicate entry in the
>> portfolio object (after running updatePortf()), where in the 2nd instance
>> there are NAs in Windows, or NaNs in Ubuntu, in:
>> -Pos.Value
>> -Period.Unrealized.PL
>> -Gross.Trading.PL
>> -Net.Trading.PL
>>
>> I *think* this is the case since I can run perTradeStats() without any
>> issues.
>>
>> Running the following after executing the code (provided below) will
>> demonstrate what I'm referring to:
>> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>>
>> I've ran it in both Windows and Ubuntu OS's, under daily and weekly
>> frequencies.
>>
>> In both OS's, daily frequencies cause checkBlotterUpdate to fail, whereas
>> under the weekly frequency, both OS's run checkBlotterUpdate successfully,
>> which allows me to generate tradeStats, and run additional functionality.
>>
>> Here's my Windows sessionInfo():
> <snip>
>>
>> and here's my Ubuntu sessionInfo():
> <snip>
>>
>> Lastly, here's the updated code, which I've attempted to reduce as much as
>> possible. Please let me know if there's any additional information that I
>> can provide.
>>
> Thank you very much for the more minimal example. This looks like a
> bug in blotter:::.updatePosPL. If you don't supply the Dates argument
> in the updatePortf call, the dates are extracted from the index of the
> Prices argument (or the object containing data for the given Symbol).
> In this case, they're extracted from the SPY object, which has an Date
> class index. The index of the posPL and txn tables are always
> POSIXct.
>
> In order to get a date range for which position P&L needs to be
> updated, we subtract a very small value from the first observation in
> Dates. Since Dates is a 'Date' vector in this case and subtracting a
> very small value causes the date to shift back an entire day. This
> causes the initializing transaction in the txn table to be included in
> the P&L calculations. For example:
> R> as.Date("1993-02-02")-0.0001
> [1] "1993-02-01"
>
> Here's a patch that seems to fix this specific issue. It needs more
> testing before I'd be comfortable committing it to the repository.
>
Brian Peterson suggested an easier/simpler fix: simply do not set the
initDate in your calls to initPortf and initAcct. I've confirmed that
avoids this issue.
> Index: updatePosPL.R
> ===================================================================
> --- updatePosPL.R (revision 1692)
> +++ updatePosPL.R (working copy)
> @@ -37,7 +37,7 @@
>
> # if no date is specified, get all available dates
> if(is.null(Dates)) {
> - Dates = index(prices)
> + Dates = as.POSIXct(index(prices))
> } else if(!is.timeBased(Dates)) {
> Dates<- if(is.na(.parseISO8601(Dates)$first.time) ||
> .parseISO8601(Dates)$first.time <
> as.POSIXct(first(index(prices)))){
>
>
>> Any thoughts and advice on how to proceed would be greatly appreciated.
>>
>> Thank you for your help.
>>
>> BEGIN CODE-----
> <snip>
>> END CODE-------
>>
>>
>>
>> Erol Biceroglu
>> erol.biceroglu at alumni.utoronto.ca
>>
>>
>> On Tue, Jul 21, 2015 at 12:49 AM, Erol Biceroglu
>> <erol.biceroglu at alumni.utoronto.ca> wrote:
>>>
>>> Hi Joshua,
>>>
>>> Thanks for the feedback, your comments are helpful.
>>>
>>> My apologies for the confusion, perhaps I provided too much information at
>>> first. I am using apply.paramset, and although it runs, it doesn't work.
>>> It will finish without an error, but the results are empty. The NA is
>>> throwing off the aggregation of the trade statistics, which occurs before I
>>> get to apply.paramset, so it would have made my code much longer.
>>>
>>> My intention, and what I was hoping I did, was identify what as causing
>>> the error (the NAs in posPL):
>>>
>>> getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>>>
>>> I tried to trim the example down, however I wasn't sure what was causing
>>> the error, so I also tried to preserve as much as I could. However I
>>> appreciate the advice, and I will keep that in mind in the future.
>>>
>>> Anyway, based on your comments, it looks like its the following rules that
>>> are causing the issue:
>>>
>>> Pct.lt.3qt.LowVol
>>>
>>> Pct.gte.3qt.LowVol
>>>
>>>
>>> These generate NA on the first time stamp, so I have something to go on to
>>> continue my investigation.
>>>
>>>
>>> Lastly, here's sessionInfo() output in case it's helpful.
>>>
>>>
>>> Thank you,
>>>
>>>
>>> > sessionInfo()
>>> R version 3.2.1 (2015-06-18)
>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>> Running under: Ubuntu 14.04.2 LTS
>>>
>>> locale:
>>> [1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
>>> LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
>>> [5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>> LC_PAPER=en_CA.UTF-8 LC_NAME=C
>>> [9] LC_ADDRESS=C LC_TELEPHONE=C
>>> LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
>>>
>>> attached base packages:
>>> [1] stats graphics grDevices utils datasets methods base
>>>
>>> other attached packages:
>>> [1] IKTrading_1.0 roxygen2_4.1.1
>>> digest_0.6.8 Rcpp_0.11.6
>>> [5] stringr_1.0.0 timeDate_3012.100
>>> quantstrat_0.9.1687 foreach_1.4.2
>>> [9] blotter_0.9.1666 PerformanceAnalytics_1.4.3541
>>> FinancialInstrument_1.2.0 quantmod_0.4-4
>>> [13] TTR_0.23-0 xts_0.9-7
>>> zoo_1.7-12
>>>
>>> loaded via a namespace (and not attached):
>>> [1] lattice_0.20-31 codetools_0.2-8 grid_3.2.1 magrittr_1.5
>>> stringi_0.5-5 iterators_1.0.7 tools_3.2.1
>>>
>>>
>>>
>>>
>>>
>>> Erol Biceroglu
>>> erol.biceroglu at alumni.utoronto.ca
>>>
>>>
>>> On Mon, Jul 20, 2015 at 11:47 PM, Joshua Ulrich <josh.m.ulrich at gmail.com>
>>> wrote:
>>>>
>>>> On Mon, Jul 20, 2015 at 7:53 PM, Erol Biceroglu
>>>> <erol.biceroglu at alumni.utoronto.ca> wrote:
>>>> > Hello,
>>>> >
>>>> > I've been playing around with quanstrat and was looking forward to
>>>> > running
>>>> > *apply.paramset()* to optimize the strategy's parameters, only to find
>>>> > an
>>>> > empty set of results.
>>>> >
>>>> Empty set of results from what? There isn't a call to apply.paramset
>>>> in your code. If you are actually running apply.paramset, please
>>>> provide the output from sessionInfo().
>>>>
>>>> > (Please note, my actual code follows after my message)
>>>> >
>>>> > After investigating, I found that *checkBlotterUpdate* fails with the
>>>> > message:
>>>> >
>>>> >> checkBlotterUpdate(tradeStrategy,tradeStrategy)
>>>> > [1] "portfolio P&L doesn't match sum of symbols P&L"
>>>> > [1] "portfolio P&L doesn't match account P&L"
>>>> > [1] FALSE
>>>> >
>>>> > Upon further investigation, I'm unable to run the following:
>>>> >> tradeStats(tradeStrategy)
>>>> > NULL
>>>> > Warning message:
>>>> > In tradeStats(tradeStrategy) : TotalNetProfit NA forSPY
>>>> >
>>>> > When I run this line (after executing everything), I find that the
>>>> > "strategy date" (the first date before the beginning of the time
>>>> > series) is
>>>> > duplicated, and in the 2nd instance, there are NAs in:
>>>> > -Pos.Value
>>>> > -Period.Unrealized.PL
>>>> > -Gross.Trading.PL
>>>> > -Net.Trading.PL
>>>> >
>>>>
>>>> > I've tried to play around with it and unfortunately I can't figure out
>>>> > what
>>>> > would cause the duplicate, generate the NA. Any thoughts or feedback
>>>> > would
>>>> > be greatly appreciated. Thanks for your help.
>>>> >
>>>> You will likely get more help if you provide a _minimal_ reproducible
>>>> example (yours didn't run for me because I don't have IKTrading
>>>> installed). It would also help to provide more detail about what
>>>> you've tried in order to solve your problem--"tried to play around
>>>> with it" doesn't help people know what you did.
>>>>
>>>> The general advice I can give you is to individually run
>>>> applyIndicators, applySignals, and applyRules; and to check the
>>>> mktdata object after you run each function.
>>>>
>>>> > Here's the code:
>>>> > ---------
>>>> > rm(list=ls())
>>>> > library(quantstrat)
>>>> > library(timeDate)
>>>> > library(stringr)
>>>> > library(IKTrading)
>>>> >
>>>> >
>>>> > Sys.setenv(TZ="UTC")
>>>> > startDate<-as.Date("1993-02-02", format="%Y-%m-%d")
>>>> > endDate<-as.Date("2015-07-17", format="%Y-%m-%d")
>>>> >
>>>> > getSymbols(Symbols = c("SPY", "^VIX")
>>>> > ,src="yahoo"
>>>> > , verbose=TRUE
>>>> > , warnings=TRUE
>>>> > , auto.assign=TRUE
>>>> > , return.class = "xts"
>>>> > , index.class = "Date"
>>>> > ,from = startDate
>>>> > , to = endDate
>>>> > )
>>>> >
>>>> > #set Financial instrument objects
>>>> > currency("USD")
>>>> > stock(primary_id = c("SPY", "VIX"),currency = "USD")
>>>> >
>>>> >
>>>> > #name
>>>> > tradeStrategy <-"SPYVIXStrategy"
>>>> >
>>>> > #Date, one day before prices
>>>> > strategyDate <- min(index(SPY)) - 1
>>>> >
>>>> >
>>>> > #rm.strat(tradeStrategy)
>>>> > #rm(mktdata)
>>>> >
>>>> > NumSh<-3
>>>> > VIXThreshold <- 20
>>>> > PctThreshold <- 0.75
>>>> >
>>>> > #init portfolio and account
>>>> > initPortf(name = tradeStrategy
>>>> > , symbols = "SPY" #as defined in Financial instrument
>>>> > , initDate = strategyDate)
>>>> >
>>>> > initAcct(name = tradeStrategy
>>>> > ,portfolios = tradeStrategy
>>>> > ,initDate = strategyDate
>>>> > ,initEq = 10e6 #as.vector(first(SPY$SPY.Close))*NumSh
>>>> > )
>>>> >
>>>> >
>>>> >
>>>> > #order book, and strategy
>>>> > initOrders(portfolio = tradeStrategy
>>>> > , initDate = strategyDate
>>>> > )
>>>> >
>>>> > #position limits
>>>> > addPosLimit(tradeStrategy, symbol = "SPY", strategyDate, maxpos =
>>>> > NumSh,
>>>> > longlevels = NumSh)
>>>> >
>>>> > strategy( tradeStrategy, store = TRUE)
>>>> >
>>>> >
>>>> >
>>>> > #define indicator function
>>>> > PctStrat <- function(x){
>>>> > data<-merge(
>>>> > runPercentRank(x=ifelse(diff(x)<0,0,diff(x)),cumulative = FALSE
>>>> > )
>>>> > , ifelse(diff(x)<0,0,diff(x))
>>>> > , x
>>>> > )
>>>> > names(data) <- c("Percentile", "PositiveDiffs","VIX.Close")
>>>> >
>>>> > data<-xts(x = sapply(data
>>>> > ,function(x){ifelse(is.na(x),0,x)}
>>>> > ), order.by = index(data)
>>>>
>>>> > )
>>>> >
>>>> > return(data)
>>>> > }
>>>> >
>>>> >
>>>> > #add indicator
>>>> > add.indicator(strategy = tradeStrategy
>>>> > , name = "PctStrat"
>>>> > , arguments = list( x = quote(VIX$VIX.Close))
>>>> > , label = "VIXPct"
>>>> > )
>>>> >
>>>> >
>>>> > # >=75th percentile move
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigThreshold"
>>>> > , arguments = list(column = c("Percentile.VIXPct")
>>>> > , threshold = quote(PctThreshold) #0.75
>>>> > , relationship = "gte"
>>>> > , cross = FALSE
>>>> > )
>>>> > , label = "Pct.gte.3Qt"
>>>> > )
>>>> >
>>>> > #<75th percentile move
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigThreshold"
>>>> > , arguments = list(column = c("Percentile.VIXPct")
>>>> > , threshold = quote(PctThreshold) #0.75
>>>> > , relationship = "lt"
>>>> > , cross = FALSE
>>>> > )
>>>> > , label = "Pct.lt.3Qt"
>>>> > )
>>>> >
>>>> > #>VIX 20
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigThreshold"
>>>> > , arguments = list(column = c("VIX.Close.VIXPct")
>>>> > , threshold = quote(VIXThreshold) #20
>>>> > , relationship = "gt"
>>>> > , cross = FALSE
>>>> > )
>>>> > , label = "HighVolatility"
>>>> > )
>>>> >
>>>> >
>>>> > #<=VIX 20
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigThreshold"
>>>> > , arguments = list(column = c("VIX.Close.VIXPct")
>>>> > , threshold = quote(VIXThreshold) #20
>>>> > , relationship = "lte"
>>>> > , cross = FALSE
>>>> > )
>>>> > , label = "LowVolatility"
>>>> > )
>>>> >
>>>> > #intersect signals
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigAND"
>>>> > , arguments = list(columns = c("Pct.lt.3Qt",
>>>> > "LowVolatility"),
>>>> > cross = TRUE)
>>>> > , label = "Pct.lt.3qt.LowVol"
>>>> > )
>>>> >
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigAND"
>>>> > , arguments = list(columns = c("Pct.lt.3Qt",
>>>> > "HighVolatility"),
>>>> > cross = FALSE)
>>>> > , label = "Pct.lt.3qt.HighVol"
>>>> > )
>>>> >
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigAND"
>>>> > , arguments = list(columns = c("Pct.gte.3Qt",
>>>> > "HighVolatility"),
>>>> > cross = FALSE)
>>>> > , label = "Pct.gte.3qt.HighVol"
>>>> > )
>>>> >
>>>> > add.signal(strategy = tradeStrategy
>>>> > , name = "sigAND"
>>>> > , arguments = list(columns = c("Pct.gte.3Qt",
>>>> > "LowVolatility"),
>>>> > cross = TRUE)
>>>> > , label = "Pct.gte.3qt.LowVol"
>>>> > )
>>>> >
>>>> > #rules
>>>> > add.rule(strategy = tradeStrategy
>>>> > , name = "ruleSignal"
>>>> > , arguments = list(sigcol="Pct.lt.3qt.LowVol"
>>>> > , sigval=TRUE
>>>> > , orderqty=NumSh
>>>> > , ordertype="market"
>>>> > , orderside=NULL#"long"
>>>> > , osFUN = "osMaxPos"
>>>> > )
>>>> > , type = "enter"
>>>> > )
>>>> >
>>>> > add.rule(strategy = tradeStrategy
>>>> > , name = "ruleSignal"
>>>> > , arguments = list(sigcol= "HighVolatility"
>>>> > #"Pct.lt.3qt.HighVol"
>>>> > , sigval=TRUE
>>>> > , orderqty="all"
>>>> > , ordertype="market"
>>>> > , orderside=NULL#"long"
>>>> > , osFUN = "osMaxPos"
>>>> > )
>>>> > , type = "exit"
>>>> > )
>>>> >
>>>> > add.rule(strategy = tradeStrategy
>>>> > , name = "ruleSignal"
>>>> > , arguments = list(sigcol="Pct.gte.3qt.LowVol"
>>>> > , sigval=TRUE
>>>> > , orderqty=1
>>>> > , ordertype="market"
>>>> > , orderside=NULL#"long"
>>>> > , osFUN = "osMaxPos"
>>>> > )
>>>> > , type = "exit"
>>>> > )
>>>> >
>>>> >
>>>> > applyStrategy(strategy = tradeStrategy
>>>> > , portfolios = tradeStrategy
>>>> > )
>>>> >
>>>> > updatePortf(tradeStrategy)
>>>> > updateAcct(tradeStrategy)
>>>> > updateEndEq(tradeStrategy)
>>>> >
>>>> >
>>>> > ###From Guy Yollin's Slides
>>>> > checkBlotterUpdate <- function(port.st,account.st,verbose=TRUE)
>>>> > {
>>>> > ok <- TRUE
>>>> > p <- getPortfolio(port.st)
>>>> > a <- getAccount(account.st)
>>>> > syms <- names(p$symbols)
>>>> > port.tot <- sum(sapply(syms,FUN = function(x) eval(parse(
>>>> >
>>>> > text=paste("sum(p$symbols",x,"posPL.USD$Net.Trading.PL)",sep="$")))))
>>>> > port.sum.tot <- sum(p$summary$Net.Trading.PL)
>>>>
>>>> > if( !isTRUE(all.equal(port.tot,port.sum.tot)) ) {
>>>> > ok <- FALSE
>>>> > if( verbose )
>>>> > print("portfolio P&L doesn't match sum of symbols P&L")
>>>> > }
>>>> > initEq <- as.numeric(first(a$summary$End.Eq))
>>>> > endEq <- as.numeric(last(a$summary$End.Eq))
>>>> > if( !isTRUE(all.equal(port.tot,endEq-initEq)) ) {
>>>> > ok <- FALSE
>>>> > if( verbose )
>>>> > print("portfolio P&L doesn't match account P&L")
>>>> > }
>>>> > if( sum(duplicated(index(p$summary))) ) {
>>>> > ok <- FALSE
>>>> > if( verbose )
>>>> > print("duplicate timestamps in portfolio summary")
>>>> > }
>>>> > if( sum(duplicated(index(a$summary))) ) {
>>>> > ok <- FALSE
>>>> > if( verbose )
>>>> > print("duplicate timestamps in account summary")
>>>> > }
>>>> > return(ok)
>>>> > }
>>>> > ###End Guy Yollin's code
>>>> >
>>>> > #This fails
>>>> > checkBlotterUpdate(tradeStrategy,tradeStrategy)
>>>> >
>>>> > chart.Posn(tradeStrategy
>>>> > , Symbol = "SPY"
>>>> > #, Dates = "1994::"
>>>> > #, Dates = "2012::"
>>>> > ,TA = "add_TA(VIX$VIX.Close)"
>>>> >
>>>> > )
>>>> >
>>>> > #Here's an error
>>>> > tradeStats(tradeStrategy)
>>>> >
>>>> > #Here's the source of the error, an NA on the 2nd line
>>>> > getPortfolio(tradeStrategy)$symbols$SPY$posPL[1:10]
>>>> >
>>>> > -------
>>>> > End code
>>>> >
>>>> >
>>>> > Erol Biceroglu
>>>> >
>>>> >
>>>> > *erol.biceroglu at alumni.utoronto.ca <erol.biceroglu at alumni.utoronto.ca>*
>>>> >
>>>> > [[alternative HTML version deleted]]
>>>> >
>>>> > _______________________________________________
>>>> > R-SIG-Finance at r-project.org mailing list
>>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>>> > -- Subscriber-posting only. If you want to post, subscribe first.
>>>> > -- Also note that this is not the r-help list where general R questions
>>>> > should go.
>>>>
>>>>
>>>>
>>>> --
>>>> Joshua Ulrich | about.me/joshuaulrich
>>>> FOSS Trading | www.fosstrading.com
>>>
>>>
>>
>
>
>
> --
> Joshua Ulrich | about.me/joshuaulrich
> FOSS Trading | www.fosstrading.com
--
Joshua Ulrich | about.me/joshuaulrich
FOSS Trading | www.fosstrading.com
More information about the R-SIG-Finance
mailing list