[R] need help with smooth.spline

[Prof Brian Ripley]

On Thu, 4 Mar 2004, Liaw, Andy wrote:

> The strange thing (to me, and to Carlisle) is the behavior of
> predict.smooth.spline() on the data he posted:  part of the predictions are
> way off from the data.  If one just plot prediction at the input data, it
> looks just fine.  What am I missing?

This is what can happen if lambda is estimated as far too small.

It is not clear to me that lambda should be being estimated and if it 
should be, that GCV is preferable to CV.  Also, there can be multiple 
local minima to either the GCV or CV criteria, and you may well want to 
set control.spar to control the range of search (which by default is a 
large range, perhaps over-large).

I was also pointing out that if the true `curve' is not smooth and you 
estimate lambda, you are not likely to get a sensible answer (for the 
reason Roger says).

But `I used smooth.spline()' without telling us how and why the particular 
arguments were chosen is exactly the sort of thing the posting guide warns 
against.

> 
> Andy
> 
> > From: Roger Koenker [mailto:roger at ysidro.econ.uiuc.edu] 
> > 
> > If one repeats the experiments in Craven and Wahba, the paper that
> > "invented" GCV you find, or at least I found, when I tried to do this
> > some years ago,  that GCV fails in about 10%
> > of cases rather catastrophically, and this is a fairly innocuous
> > setting.  So one way to interpret Brian's comment would be that maybe
> > it is GCV that is failing, and another choice of lambda might 
> > do better.
> > 
> > Obviously, Brian can interpret for himself.  I would only add 
> > that in cases
> > where you really want something with sharp breaks in derivatives then
> > then the usual L_2 roughness penalties are not very 
> > appropriate however
> > you choose to do the smoothing.
> > 
> > url:	www.econ.uiuc.edu/~roger/my.html	Roger Koenker
> > email	rkoenker at uiuc.edu			Department of Economics
> > vox: 	217-333-4558				University of Illinois
> > fax:   	217-244-6678				
> > Champaign, IL 61820
> > 
> > 
> > 
> > 
> 
> 
> ------------------------------------------------------------------------------
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New
> Jersey, USA 08889), and/or its affiliates (which may be known outside the
> United States as Merck Frosst, Merck Sharp & Dohme or MSD and in Japan as
> Banyu) that may be confidential, proprietary copyrighted and/or legally
> privileged. It is intended solely for the use of the individual or entity
> named on this message.  If you are not the intended recipient, and have
> received this message in error, please notify us immediately by reply e-mail
> and then delete it from your system.
> ------------------------------------------------------------------------------
> 
> 

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595