[R-SIG-Finance] Stability of trading models

[Brian G. Peterson]

Gero Schwenk wrote:
> <... snip ...>
> In order to assess the stability of out-of-sample fit of a given 
> model, I would normally draw cross-validation samples and partition 
> them into training- and test subsets. Grounds for this would be the 
> assumption of independent observations contained in the model and 
> forced onto the data by backshifting them.
>
> However, I'm reluctant to believe that the data-generating process 
> doesn't change over time, which is implied by my procedures. If this 
> was true and time was not an issue, it should not be necessary to 
> recalibrate the model, even after a long period of out-of-sample 
> prediction. This seems overly optimistic to me.
>
> Returning to the question of stability assessment and 
> cross-validation, I would like to know if there is some pragmatic 
> solution. Is simple cross-validation viable? Do I need to go far into 
> the past using some possibly sliding training- and test-windows? Or 
> has anybody a different suggestion how to deal with this problem in 
> the realm of regression models?
While we would all like a perfectly stable model, the reality is usually different.

Since you've said that your model is regression-based, take a look at chart.RollingRegression.  This will let you see the stability of your regression model over different rolling windows.  I suggest checking longer, shorter, and from-inception windows.

>From there, you'll have a lot of information to refine your modeling/tuning approach.

I generally am dubious that financial time series or trading models are completely stable over long periods of time.

Cheers,

    - Brian

-- 
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock