[R-SIG-Finance] Fwd: model confidence sets in R

[Stefan Janse van Rensburg]

There are at least three different packages implementing the Model
Confidence Set (MCS) procedure of Hansen, Lunde and Nason (2011). These
three packages, however, produce very different MCS p-values and rather
different superior model sets.

To illustrate, I'm including the following minimal working example. Code is
also available at the following link:
https://gist.github.com/sjvrensburg/57c325f62c52b9df56c82ca41f81de28

#####################################################
require(MCS)
require(rugarch)
require(modelconf)
require(parallel)
require(dplyr)

cl <- makeForkCluster(nnodes = 8)   # Running Linux, so can fork.
data(Loss)        # Loss data from the MCS package
loss_mat <- as.matrix(Loss[, 1:10]) # mcsTest requires a "plain" matrix
model_names <- colnames(loss_mat)

# Parameters for functions...
alpha <- 0.2
B <- 50000        # Make this big, in order to reduce the variability.
block_len <- 33   # Block length calculated as in MCS package.

# Calculate MCS using various packages. Takes a while...
set.seed(123456)
rugarch_pkg <- mcsTest(losses = loss_mat, alpha = alpha, nboot = B,
                       nblock = block_len, boot = "block")
set.seed(123456)
mcs_pkg <- MCSprocedure(Loss = loss_mat, alpha = alpha, B = B,
                        statistic = 'TR', k = block_len, cl = cl)
set.seed(123456)
modelconf_pkg <- estMCS(loss = loss_mat, test = "t.range", B = B,
                        l = block_len)
stopCluster()

# Extract and format results for comparison.
rugarch_results <- cbind(Model = model_names[c(rugarch_pkg$excludedR,
                                               rugarch_pkg$includedR)],
                         pval = rugarch_pkg$pvalsR)

mcs_results <- mcs_pkg using show %>%
  as.data.frame %>%
  cbind(Model = row.names(mcs_pkg using show), .) %>%
  select(Model, MCS_R) %>%
  arrange(MCS_R)

modelconf_results <- modelconf_pkg %>%
  as.data.frame %>%
  cbind(Model = row.names(modelconf_pkg), .) %>%
  select(Model, `MCS p-val`) %>%
  arrange(`MCS p-val`)

# Inspect results
#   rugarch: Only sGARCH-ghyp & sGARCH-sged in SSM
print(tail(rugarch_results, 5))
# Model
# [6,] "sGARCH-sstd"  "0.06714"
# [7,] "sGARCH-snorm" "0.06714"
# [8,] "sGARCH-jsu"   "0.06714"
# [9,] "sGARCH-ghyp"  "0.33244"
# [10,] "sGARCH-sged"  "1"

#   MCS: only returns SSM, which includes 7 models.
print(tail(mcs_results, 5))
# 3  sGARCH-ghyp 0.82144
# 4   sGARCH-ged 0.96256
# 5   eGARCH-std 0.96704
# 6 sGARCH-snorm 0.99584
# 7  sGARCH-sged 1.00000

#   modelconf: SSM includes 7 models.
print(tail(modelconf_results, 5))
# Model MCS p-val
# 6   sGARCH-ghyp   0.70332
# 7    sGARCH-ged   0.87124
# 8  sGARCH-snorm   0.87124
# 9    eGARCH-std   0.87124
# 10  sGARCH-sged   1.00000
#######################################################

In all cases, p-values are based on the range statistic T_R.

The superior set of models (SSM) produced by the MCS and modelconf packages
are the same, but the MCS p-values are quite different. The rank of models
in the SSM are also different. The SSM produced by the rugarch package
differs drastically, and contains only two models.

Differences in p-values are to be expected, since these procedures are
based on the bootstrap. However, given the large number of bootstrap
replications, surely the differences should be small? Also, notice the
repetition or ties in p-values with the results from mcsTest and estMCS.

Am I doing something wrong or misinterpreting the documentation? Does any
one else use any of these procedures? Which do you prefer and recommend? I
was, initially, planning on using the MCS package. But I don't like that it
only returns the SSM. That's when I started looking at alternatives, and
now I'm totally confused!

Any help, whatsoever, would be greatly appreciated.

Kind regards,

Stéfan Janse van Rensburg
-----
Ph.D. Candidate, Mathematical Statistics, NMU

LinkedIn:
http://www.linkedin.com/profile/public-profile-settings?trk=prof-edit-edit-public_profile

My photography website: http://www.sjvrensburg.com/

	[[alternative HTML version deleted]]