[Rd] Bug in nlm, found using sem; failure in several flavors (PR#13883)

[Peter Dalgaard]

William Dunlap wrote:

> Running his example under valgrind (a memory misuse checker on Linux)
> does show it using memory it should not be using in the optimization C
> code
> around where it is copying the gradient vector.  The 
> 
> ==10916== Invalid read of size 1
> ==10916==    at 0x400686D: memcpy (mc_replace_strmem.c:406)
> ==10916==    by 0x8072BC1: FT_store (optimize.c:319)
> ==10916==    by 0x8072F92: fcn (optimize.c:408)

Looks like this does the trick (you do want to protect "s" because 
install() may allocate):

[pd at titmouse2 R]$ svn diff
Index: src/main/optimize.c
===================================================================
--- src/main/optimize.c	(revision 49168)
+++ src/main/optimize.c	(working copy)
@@ -378,7 +378,7 @@
  	if (!R_FINITE(x[i])) error(_("non-finite value supplied by 'nlm'"));
  	REAL(s)[i] = x[i];
      }
-    s = eval(state->R_fcall, state->R_env);
+    s = PROTECT(eval(state->R_fcall, state->R_env));
      switch(TYPEOF(s)) {
      case INTSXP:
  	if (length(s) != 1) goto badvalue;
@@ -400,13 +400,14 @@
  	goto badvalue;
      }
      if (state->have_gradient) {
-	g = REAL(coerceVector(getAttrib(s, install("gradient")), REALSXP));
+	g = REAL(PROTECT(coerceVector(getAttrib(s, install("gradient")), 
REALSXP)));
  	if (state->have_hessian) {
-	    h = REAL(coerceVector(getAttrib(s, install("hessian")), REALSXP));
+	    h = REAL(PROTECT(coerceVector(getAttrib(s, install("hessian")), 
REALSXP)));
  	}
      }
      FT_store(n, *f, x, g, h, state);
-    return;
+    UNPROTECT(1 + state->have_gradient + state->have_hessian);
+    return;

   badvalue:
      error(_("invalid function value in 'nlm' optimizer"));


-- 
    O__  ---- Peter Dalgaard             Øster Farimagsgade 5, Entr.B
   c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
  (*) \(*) -- University of Copenhagen   Denmark      Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)              FAX: (+45) 35327907