[R] Numerical precision and signif()

[Don MacQueen]

I have encountered something about signif() that I don't understand. 
It presents a problem for the situation in which I'm using it.

>  foo <- 3.7e7
>  formatC(foo,digits=22)
[1] "               37000000"
>  formatC(signif(foo,2),digits=22)
[1] "36999999.99999997764826"
>  formatC(signif(foo,3),digits=22)
[1] "36999999.99999997764826"
>  formatC(signif(foo,4),digits=22)
[1] "36999999.99999998509884"
>  formatC(signif(foo,5),digits=22)
[1] "36999999.99999999254942"
>  formatC(signif(foo,6),digits=22)
[1] "36999999.99999999254942"
>  formatC(signif(foo,7),digits=22)
[1] "               37000000"

The difference between foo and signif(foo,5) is large enough to make 
a practical difference:

>  signif(5.5e-7*foo,3)
[1] 20.4
>  signif(5.5e-7*signif(foo,5),3)
[1] 20.3

>  round(5.5e-7*foo,1)
[1] 20.4
>  round(5.5e-7*signif(foo,5),1)
[1] 20.3


(foo is a unit conversion factor; I shouldn't need to use signif() on 
it, but I was trying it as a workaround for another problem, and this 
came up. 5.5e-7 is a measurement whose units need to be converted)


While I'm at it, why is this?
>  format(0.037,dig=22)
[1] "0.037"
>  formatC(0.037,dig=22)
[1] "0.03699999999999999816813"



>  version
          _
platform sparc-sun-solaris2.7
arch     sparc
os       solaris2.7
system   sparc, solaris2.7
status
major    1
minor    6.2
year     2003
month    01
day      10
language R

Thanks
-Don
-- 
--------------------------------------
Don MacQueen
Environmental Protection Department
Lawrence Livermore National Laboratory
Livermore, CA, USA
--------------------------------------