[R] How to improve the robustness of "loess"? - example included.

[Emmanuel Levy]

Hi,

I posted a message earlier entitled "How to fit a line through the
"Mountain crest" ..."

I figured loess is probably the best way, but it seems that the
problem is the robustness of the fit. Below I paste an example to
illustrate the problem:

    tmp=rnorm(2000)
    X.background = 5+tmp; Y.background = 5+ (10*tmp+rnorm(2000))
    X.specific = 3.5+3*runif(1000); Y.specific = 5+120*runif(1000)
    X = c(X.background, X.specific);Y = c(Y.background, Y.specific)
    MINx=range(X)[1];MAXx=range(X)[2]

    my.loess = loess(Y ~ X, data.frame( X = X, Y = Y),
family="symmetric", degree=2, span=0.1)
    lo.pred = predict(my.loess, data.frame(X = seq(MINx, MAXx,
length=100)), se=TRUE)
    plot( seq(MINx, MAXx, length=100), lo.pred$fit, lwd=2,col=2, "l")
    points(X,Y, col= grey(abs(my.loess$res)/max(abs(my.loess$res))) )

As you will see, the red line does not follow the "background" signal.
However, when decreasing the "specific" signal to 500 points it
becomes perfect.

I'm sure there is a way to "tune" the fitting so that it works but I
can't figure out how. Importantly, *I cannot increase the span*
because in reality the relationship I'm looking at is more complex so
I need a small  span value to allow for a close fit.

I foresee that changing the "weigthing" is the way to go but I do not
really understand how the "weight" option is used (I tried to change
it and nothing happened), and also the embedded "tricubic weighting"
does not seem changeable.

So any idea would be very welcome.

Emmanuel