[Rd] Issues of R_pretty in src/appl/pretty.c

[Suharto Anggono Suharto Anggono]

See https://stat.ethz.ch/pipermail/r-devel/2017-August/074746.html for the origin of the example here.

That
pretty(c(-1,1)*1e300, n = 1e9, min.n = 1) gave 20 intervals, far from 1e9, but
pretty(c(-1,1)*1e300, n = 1e6, min.n = 1) gave 1000000 intervals
(on a machine), made me trace through the code to function 'R_pretty' in https://svn.r-project.org/R/trunk/src/appl/pretty.c .

*lo is -1e300, *up is 1e300.
cell = fmax2(fabs(*lo),fabs(*up));
'cell' is 1e300.
i_small = dx < cell * U * imax2(1,*ndiv) * DBL_EPSILON *3;
When *ndiv is (int) 1e9, apparently cell * U * imax2(1,*ndiv) overflows to infinity and 'i_small' is 1 (true). It doesn't happen when *ndiv is (int) 1e6.

Putting parentheses may avoid the floating point overflow. For example,
i_small = dx < cell * (U * imax2(1,*ndiv) * DBL_EPSILON) *3;

The part
U = (1 + (h5 >= 1.5*h+.5)) ? 1/(1+h) : 1.5/(1+h5);
is strange. Because (h5 >= 1.5*h+.5) is 1 or 0, (1 + (h5 >= 1.5*h+.5)) is never zero and 1/(1+h) will always be chosen.

The comment for 'rounding_eps' says "1e-7 is consistent with seq.default()". Currently, seq.default() uses 1e-10 as fuzz.