[R-SIG-Finance] Back testing Expected Shortfall

alexios galanos @|ex|o@ @end|ng |rom 4d@c@pe@com
Mon Jun 15 17:00:47 CEST 2020


Hi Pit and thanks for sharing.

I was not aware of the Gneiting paper, but the Gneiting and Raftery 
(2007) paper discusses scoring rules and their mean interval score (MIS) 
has been used in the M4 competition (implemented in the greybox package).

Best,

Alexios

On 6/15/20 7:34 AM, Pit Götz wrote:
> Hello everyone,
> 
> I work at a university in germany and we are also currently working on 
> forecasting ES and (of course) backtesting of said forecasts.
> 
> Over the last few months some students, who are writing their masters 
> thesis at our chair, had to some litarature research.
> Thats why I wanted to give you a very brief overview of their findings:
> 
> The most widely applied ES backtests seems to be the backtest by McNeil, 
> Frey and Embrechts (2000), implemented for example in the rugarch package.
> (the test was already mentioned here by Alexios)
> 
> In addition to the already mentioned tests and the paper by Acerby and 
> Szekely I wanted to add the following:
> 
> A Hitsequence based backtest was introduced for by Du, Escanciano 
> (2017). As far as I am concerned, this test has not yet been implemented 
> in a package, but their code is available online. In a broader view, 
> this test is a special case of a spectral measure test by Costanzino, 
> Curran (2014), which was then extended to a Basel-Like traffic light 
> approach in 2018 (Not sure about the availability of code).
> 
> In Emmer et al. (2015) it is suggested, that a suitable ES forecast can 
> be approximated by only 4 different VaR forecasts. This also suggests, 
> that you can backtest ES, forecasted by a model that forecasts both, ES 
> and VaR, such as GARCH, by backtesting th 4 different VaR forecasts.
> However this approach seems to need more empirical valuation.
> 
> I also wanted to mention the paper by Gneiting (2011), showing that the 
> ES lacks elicitability property. This can lead to complications, when 
> you try to backtest the ES itself as a point forecast.However, this 
> property can be used to construct a model comparison like backtest as in 
> Fissler et al. (2015).
> 
> More reacently, a quantile regression based approach has been suggested 
> by Coupier, Leymarie (2020). I have not yet read said paper and 
> therefore I can not tell you anything about it.
> 
> I hope that this message gives you some new insights and some usefull 
> information.
> 
> Best regards,
> Pit
> 
> 
> 
> Research Associate
> 
> *Martin-Luther-Universität Halle-Wittenberg*
> 
> Chair of Finance & Banking
> 
> Große Steinstraße 73 | D-06108 Halle | Germany
> Tel 0049 345 5523452
> 
> 
>>>> Daniel Cegiełka <daniel.cegielka using gmail.com> 10.06.20 21.49 Uhr >>>
> śr., 10 cze 2020 o 21:14 alexios galanos <alexios using 4dscape.com> napisał(a):
>  >
>  >
>  >
>  > On 6/10/20 11:08 AM, Daniel Cegiełka wrote:
>  > > śr., 10 cze 2020 o 19:23 Brian G. Peterson <brian using braverock.com> 
> napisał(a):
>  > >>
>  > >> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
>  > >>> I was looking for an idea how banks backtest their models for
>  > >>> Expected
>  > >>> Shortfall. Backtesting VaR is well documented but I failed to get any
>  > >>> practical idea about backtesting ES.
>  > >>>
>  > >>> Any pointer towards the best practice will be really helpful.
>  > >>
>  > >> If you are using Normal VaR, then you know the Expected Shortfall
>  > >> estimate too.
>  > >>
>  > >> If you are using a different mechanism, then of course the mean loss
>  > >> when the loss exceeds the VaR may be significantly different than the
>  > >> Normal ES.
>  > >>
>  > >> So, to backetesting... the newest Basel standard replaces VaR with ES,
>  > >> and requires that banks justify their use of a particular ES model 
> that
>  > >> they are using to calculate required regulatory capital.
>  > >
>  > > In my opinion, there is one aspect that introduces some confusion. ES
>  > > (CVaR) is now common, but many people, perhaps out of habit, maybe for
>  > > historical reasons, still use the term VaR instead of the correct name
>  > > (ES).
>  >
>  > Not sure I follow. VaR and ES are different measures. VaR is a
>  > quantile while ES is the average loss conditional on that quantile
>  > (i.e. the expected loss conditional that the loss is greater than
>  > the quantile of the loss distribution).
> 
> I agree that these names should not be confused. However, I
> encountered that the _name_ "VaR" is used for ES. In my opinion, this
> is due to a mental shortcut, or it's a historical habit. Such
> imprecise use of the names often leads to misunderstanding.
> 
> Daniel
> 
>  > Regards,
>  >
>  > Alexios
>  >
>  > >
>  > > Best regards,
>  > > Daniel
>  > >
>  > >
>  > >> Regards,
>  > >>
>  > >> Brian
>  > >>
>  > >>
>  > >> --
>  > >> Brian G. Peterson
>  > >> ph: +1.773.459.4973
>  > >> im: bgpbraverock
>  > >>
>  > >> _______________________________________________
>  > >> R-SIG-Finance using r-project.org mailing list
>  > >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>  > >> -- Subscriber-posting only. If you want to post, subscribe first.
>  > >> -- Also note that this is not the r-help list where general R 
> questions should go.
>  > >
>  > > _______________________________________________
>  > > R-SIG-Finance using r-project.org mailing list
>  > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>  > > -- Subscriber-posting only. If you want to post, subscribe first.
>  > > -- Also note that this is not the r-help list where general R 
> questions should go.
>  > >
> 
> _______________________________________________
> R-SIG-Finance using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions 
> should go.



More information about the R-SIG-Finance mailing list