[R-SIG-Finance] portfolio.optim and error in solve.QP: matrix D not positive definite

Sun Jan 30 10:34:20 CET 2011

Krishna has a good point.  Getting the
variance matrix to be barely positive
definite is still not a good thing for
optimization.

An eigenvalue that is essentially zero
says that there is a portfolio that is
essentially riskless.  And we haven't
had any of those since subprime mortgages
fell out of favor.

As Krishna says a factor model will give
you a better result.  A good (possibly
better) alternative is a Ledoit-Wolf
shrinkage estimate.

Functions to estimate either of those are
in the 'BurStFin' package which still hasn't
arrived in CRAN.  But you can get it with:

install.packages('BurStFin', repos="http://www.burns-stat.com/R")

The 'tawny' package has a function for
Ledoit-Wolf.

On 29/01/2011 21:32, krishna wrote:
> The problem could be due to few reasons the papers by Higham & Rebonato
> are a good read as to what the recipe does. The simplest recipe is to
> set the negative eigenvalues to a small positive number and rescale the
> matrix.
>
> There are a few causes for this if you create corr matrices from
> bivariate estimation and slap them together that may not be PD. In some
> cases in derivative pricing the matrix is hand glued together from
> implied correlations between two assets. Now a matrix of such implied
> corrs does not have to be PD and fails.
>
> Sample corr matrices are by definition PD however often times if you
> have a lot of missing data(illiquid names in your universe?) and if you
> do a na.locf (creating a const asset) this could do it as well.
>
> It might be useful to do multivariate imputation with some o the R
> packages rather than deletion or carrying fwd on your data.
>
> There are other reasons besides all of the above for your corr matrix
> being non PD. I think if you are working with such a large universe it
> might be easier to have factor corr matrices using PCA or ICA. This way
> if you manage to label the factors then you can see what bets your
> optimizer is taking.
>
>
> HTH
>
> Best
> Krishna
>
>
>
> On Jan 29, 2011, at 3:52 PM, "Lui ##" <lui.r.project at googlemail.com> wrote:
>
>> Hello everybody,
>>
>> sorry for my delayed "thanks" note - I was travelling.
>>
>> @Arun: Debugging the underlying code is a little bit difficult since
>> the optimizer was written in FORTRAN. I think going for the nearest PD
>> (as Krishna also suggested) might be the best way. However, I honestly
>> don't understand why it is not PD... Does anybody have an explanation
>> for that?
>> @Guy: The weird thing is that I got the error code without the t(x) in
>> the first place. t(x) solved the problem (for some assets) and the
>> result indicated that it took a look at the assets and not the
>> observations... I am going to give it a try with the covariance matrix
>> again and let you know if it worked out... strange though.
>> @Krishna: Thanks for your link! I think that really helps... I am
>> going to try it out! Do you have an explanation why this is a common
>> problem with a large number of assets?
>>
>> Thank you! Have a nice weekend!
>>
>> Lui
>>
>> On Fri, Jan 28, 2011 at 3:51 PM, krishna <kriskumar at earthlink.net> wrote:
>>> Also the link below might help with a large number of assets this is a
>>> common problem.
>>>
>>> https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html
>>>
>>>
>>> Cheers
>>> Krishna
>>>
>>>
>>> On Jan 27, 2011, at 12:03 PM, Guy Yollin <gyollin at r-programming.org>
>>> wrote:
>>>
>>>> Hi Lui,
>>>>
>>>> Without seeing the data this is just speculation but...
>>>>
>>>> Are you sure you want t(x)? If you're mixing up your observations
>>>> versus
>>>> your assets this may explain the error.
>>>>
>>>> The first parameter of portfolio.optim (in the tseries package) is a
>>>> returns matrix, one column for each asset and one row for each day
>>>> (assuming
>>>> daily returns). If you have this wrong then for your small datasets
>>>> you'd
>>>> have more columns than rows and this could produce that error.
>>>>
>>>> Also, you don't have to pass the entire returns matrix to
>>>> portfolio.optim,
>>>> you could pass just the covariance matrix you calculate yourself and a
>>>> vector (1-row matrix) of mean returns as follows:
>>>>
>>>>
>>>> library(tseries)
>>>> set.seed(2)
>>>> R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100
>>>> observations
>>>> averet <- matrix(apply(R,2,mean),nrow=1)
>>>> rcov <- cov(R)
>>>> current_er <- 0.05
>>>> (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless =
>>>> FALSE,shorts = FALSE, rf = 0.0))
>>>>
>>>> Hope this helps.
>>>>
>>>> Best,
>>>>
>>>> Guy
>>>>
>>>>
>>>> On 1/26/2011 7:51 PM, Lui ## wrote:
>>>>>
>>>>> Dear Group,
>>>>>
>>>>> I have a large set of stocks and want to determine the efficient
>>>>> frontier. The data set covers approx. 1.5 years and S&P 500 companies
>>>>> (nothing weird). portfolio.optim from the PerformanceAnalytics package
>>>>> works very well and fast. However, whenever I decrease the number of
>>>>> stocks in the portfolio (to 10 or 400), I receive an error message:
>>>>>
>>>>> "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) :
>>>>> matrix D in quadratic function is not positive definite!"
>>>>>
>>>>> My command settings for portfolio.optim were:
>>>>>
>>>>> seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE,
>>>>> shorts = FALSE, rf = 0.0)
>>>>>
>>>>> Even when I tried it with shorts = TRUE the error would still remain.
>>>>> x is the set of stocks (stocks in columns, time in rows), current_er
>>>>> is the target return (lies between the minimal mean and the maximum
>>>>> mean of a long only portfolio).
>>>>> I can not post the stock data here - so maybe you have some general
>>>>> suggestions for me of what could have gone wrong... The covariance
>>>>> matrix is positive definite. What could cause the problem? It works
>>>>> fine with the large data set but does not work at all with the small
>>>>> one...
>>>>> Thanks a lot for your suggestions!
>>>>>
>>>>> Lui
>>>>>
>>>>> _______________________________________________
>>>>> R-SIG-Finance at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>>>> -- Subscriber-posting only. If you want to post, subscribe first.
>>>>> -- Also note that this is not the r-help list where general R
>>>>> questions
>>>>> should go.
>>>>>
>>>>
>>>> _______________________________________________
>>>> R-SIG-Finance at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>>> -- Subscriber-posting only. If you want to post, subscribe first.
>>>> -- Also note that this is not the r-help list where general R questions
>>>> should go.
>>>
>>> _______________________________________________
>>> R-SIG-Finance at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>> -- Subscriber-posting only. If you want to post, subscribe first.
>>> -- Also note that this is not the r-help list where general R questions
>>> should go.
>>>
>>
>> _______________________________________________
>> R-SIG-Finance at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R
>> questions should go.
>
> _______________________________________________
> R-SIG-Finance at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

-- 
Patrick Burns
patrick at burns-stat.com
http://www.burns-stat.com
http://www.portfolioprobe.com/blog
twitter: @portfolioprobe