Hi All,

Changing the data type to INT4U worked for my application (14400 rows and
19200 columns).

Thank you all for your help!


On Tue, Apr 30, 2013 at 1:30 PM, Robert J. Hijmans <r.hijmans@gmail.com>wrote:

> This is indeed because of precision loss due to FLT4S. Numbers larger
> than 16777216 cannot be recovered to the exact integer from a 4 byte
> floating point file. To avoid this I will change the default that
> 'raster' uses to FLT8S. This will double file sizes if the default is
> used.
>
> Ted, you can achieve what you are doing in this way, using the "init"
> function, using the "datatype="  argument:
>
> library(raster)
> nColVal<-1900
> outFileName <- 'tmp3.grd'
> out <- raster(nrow=20000,ncol=nColVal)#modisRasterAll
>
> out <- init(out, v='cell', filename=outFileName, datatype='INT4S')
>
> # or    out <- init(out, v='cell', filename=outFileName, datatype='FLT8S')
>
>
> Robert
>
> On Tue, Apr 30, 2013 at 9:55 AM, Matteo Mattiuzzi
> <matteo.mattiuzzi@boku.ac.at> wrote:
> > Hi Tom
> >
> >
> > In fact there is a difference:
> > see ?dataType why I do not use 'INT4U'. Probably the reason is that
> writeStart (see ?writeRaster) uses FLT4S as default and adding values in
> blocks the function doesn't know that FLT4S is not enough where else with
> out[] <- 1:ncell(out) it knows! I think that the solution.
> >
> >
> >> ?dataType
> >> nColVal<-1900
> >> outFileName <- 'tmp.grd'
> >>
> >> #Set up raster
> >> out <- raster(nrow=20000,ncol=nColVal)#modisRasterAll
> >>
> >> ##Get number of colums and blocksize
> >> outCol <- ncol(out)
> >> bs <- blockSize(out)
> >>
> >> ##Write to file
> >> out <- writeStart(out,outFileName,overwrite=TRUE,datatype="FLT8S")
> >> for(i in 1:bs$n){
> > + out <-
> > + writeValues(out,cellFromRowCol(out,bs$row[i],1) :
> > + cellFromRowCol(out,bs$row[i] + bs$nrow[i] -1 , ncol(out)) ,bs$row[i])
> > + }
> >> out <- writeStop(out)
> >>
> >> out2 <- raster(out)
> >> out2[]<- 1:ncell(out2)
> >>
> >> res <- out-out2
> > tail(res)
> >       1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893
> 1894
> > 19991    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19992    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19993    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19994    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19995    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19996    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19997    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19998    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 19999    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> > 20000    0    0    0    0    0    0    0    0    0    0    0    0    0
>  0
> >       1895 1896 1897 1898 1899 1900
> > 19991    0    0    0    0    0    0
> > 19992    0    0    0    0    0    0
> > 19993    0    0    0    0    0    0
> > 19994    0    0    0    0    0    0
> > 19995    0    0    0    0    0    0
> > 19996    0    0    0    0    0    0
> > 19997    0    0    0    0    0    0
> > 19998    0    0    0    0    0    0
> > 19999    0    0    0    0    0    0
> > 20000    0    0    0    0    0    0
> >> tail(out)
> >           1881     1882     1883     1884     1885     1886     1887
> 1888
> > 19991 37982881 37982882 37982883 37982884 37982885 37982886 37982887
> 37982888
> > 19992 37984781 37984782 37984783 37984784 37984785 37984786 37984787
> 37984788
> > 19993 37986681 37986682 37986683 37986684 37986685 37986686 37986687
> 37986688
> > 19994 37988581 37988582 37988583 37988584 37988585 37988586 37988587
> 37988588
> > 19995 37990481 37990482 37990483 37990484 37990485 37990486 37990487
> 37990488
> > 19996 37992381 37992382 37992383 37992384 37992385 37992386 37992387
> 37992388
> > 19997 37994281 37994282 37994283 37994284 37994285 37994286 37994287
> 37994288
> > 19998 37996181 37996182 37996183 37996184 37996185 37996186 37996187
> 37996188
> > 19999 37998081 37998082 37998083 37998084 37998085 37998086 37998087
> 37998088
> > 20000 37999981 37999982 37999983 37999984 37999985 37999986 37999987
> 37999988
> >           1889     1890     1891     1892     1893     1894     1895
> 1896
> > 19991 37982889 37982890 37982891 37982892 37982893 37982894 37982895
> 37982896
> > 19992 37984789 37984790 37984791 37984792 37984793 37984794 37984795
> 37984796
> > 19993 37986689 37986690 37986691 37986692 37986693 37986694 37986695
> 37986696
> > 19994 37988589 37988590 37988591 37988592 37988593 37988594 37988595
> 37988596
> > 19995 37990489 37990490 37990491 37990492 37990493 37990494 37990495
> 37990496
> > 19996 37992389 37992390 37992391 37992392 37992393 37992394 37992395
> 37992396
> > 19997 37994289 37994290 37994291 37994292 37994293 37994294 37994295
> 37994296
> > 19998 37996189 37996190 37996191 37996192 37996193 37996194 37996195
> 37996196
> > 19999 37998089 37998090 37998091 37998092 37998093 37998094 37998095
> 37998096
> > 20000 37999989 37999990 37999991 37999992 37999993 37999994 37999995
> 37999996
> >           1897     1898     1899     1900
> > 19991 37982897 37982898 37982899 37982900
> > 19992 37984797 37984798 37984799 37984800
> > 19993 37986697 37986698 37986699 37986700
> > 19994 37988597 37988598 37988599 37988600
> > 19995 37990497 37990498 37990499 37990500
> > 19996 37992397 37992398 37992399 37992400
> > 19997 37994297 37994298 37994299 37994300
> > 19998 37996197 37996198 37996199 37996200
> > 19999 37998097 37998098 37998099 37998100
> > 20000 37999997 37999998 37999999 38000000
> >
> >
> >>>> Tom Philippi  04/30/13 6:27 PM >>>
> > Ted & Matteo--
> > Is this perhaps a dataType problem?  I don't recall the specifics of the
> > significand of FLT4S but it's roughly 8 digits, so 38000000 may exceed
> it.
> >  You might try changing the dataType to INT4U to see if the dataType is
> the
> > problem.
> >
> > Tom 2
> >
> >
> >
> > On Tue, Apr 30, 2013 at 7:22 AM, Matteo Mattiuzzi <
> > matteo.mattiuzzi@boku.ac.at> wrote:
> >
> >> Ted you are right!
> >>
> >>
> >> I have tried a few things and I can not explain the behaviour.
> >> @Robert may the example below helps a little further.
> >> I have also tried to write into a tif and an img file but it doesn't
> >> change. Also after reloading the file out <- raster("tmp.grd") the
> problem
> >> persists. Starting at row 8831 the values are strange...
> >> [I have also changed the value generation part, as it is little bit
> >> clearer to me for reading: cellFromRowCol(out,bs$row[i],1) :
> >> cellFromRowCol(out,bs$row[i] + bs$nrow[i] -1 , ncol(out)) ]
> >>
> >>
> >> R version 2.15.3 (2013-03-01) -- "Security Blanket"
> >> Copyright (C) 2013 The R Foundation for Statistical Computing
> >> ISBN 3-900051-07-0
> >> Platform: x86_64-pc-linux-gnu (64-bit)
> >>
> >>
> >> R is free software and comes with ABSOLUTELY NO WARRANTY.
> >> You are welcome to redistribute it under certain conditions.
> >> Type 'license()' or 'licence()' for distribution details.
> >>
> >>
> >>   Natural language support but running in an English locale
> >>
> >>
> >> R is a collaborative project with many contributors.
> >> Type 'contributors()' for more information and
> >> 'citation()' on how to cite R or R packages in publications.
> >>
> >>
> >> Type 'demo()' for some demos, 'help()' for on-line help, or
> >> 'help.start()' for an HTML browser interface to help.
> >> Type 'q()' to quit R.
> >>
> >>
> >> > library(raster)
> >> Loading required package: sp
> >> > sessionInfo()
> >> R version 2.15.3 (2013-03-01)
> >> Platform: x86_64-pc-linux-gnu (64-bit)
> >>
> >>
> >> locale:
> >>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> >>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> >>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> >>  [7] LC_PAPER=C                 LC_NAME=C
> >>  [9] LC_ADDRESS=C               LC_TELEPHONE=C
> >> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >>
> >>
> >> attached base packages:
> >> [1] stats     graphics  grDevices utils     datasets  methods   base
> >>
> >>
> >> other attached packages:
> >> [1] raster_2.1-25 sp_1.0-9
> >>
> >>
> >> loaded via a namespace (and not attached):
> >> [1] grid_2.15.3     lattice_0.20-15
> >> >
> >> > ##Write tile indexed MODIS file
> >> > #Prelims
> >> > nColVal<-1900
> >> > outFileName <- 'tmp.grd'
> >> >
> >> > #Set up raster
> >> > out <- raster(nrow=20000,ncol=nColVal)#modisRasterAll
> >> >
> >> > ##Get number of colums and blocksize
> >> > outCol <- ncol(out)
> >> > bs <- blockSize(out)
> >> >
> >> > ##Write to file
> >> > out <- writeStart(out,outFileName,overwrite=TRUE)
> >> > for(i in 1:bs$n){
> >> + out <-
> >> + writeValues(out,cellFromRowCol(out,bs$row[i],1) :
> >> cellFromRowCol(out,bs$row[i] + bs$nrow[i] -1 , ncol(out)) ,bs$row[i])
> >> + }
> >> > out <- writeStop(out)
> >> >
> >> > out2 <- raster(out)
> >> > out2[]<- 1:ncell(out2)
> >> >
> >> > res <- out-out2
> >> rgdal: version: 0.8-8, (SVN revision 463)
> >> Geospatial Data Abstraction Library extensions to R successfully loaded
> >> Loaded GDAL runtime: GDAL 1.9.2, released 2012/10/08
> >> Path to GDAL shared files: /usr/share/gdal/1.9
> >> Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
> >> Path to PROJ.4 shared files: (autodetected)
> >> >
> >> > sum(res[])
> >> [1] 0
> >> >
> >> > tail(out)
> >>           1881     1882     1883     1884     1885     1886     1887
> >> 1888
> >> 19991 37982880 37982880 37982884 37982884 37982884 37982888 37982888
> >> 37982888
> >> 19992 37984780 37984784 37984784 37984784 37984784 37984784 37984788
> >> 37984788
> >> 19993 37986680 37986680 37986684 37986684 37986684 37986688 37986688
> >> 37986688
> >> 19994 37988580 37988584 37988584 37988584 37988584 37988584 37988588
> >> 37988588
> >> 19995 37990480 37990480 37990484 37990484 37990484 37990488 37990488
> >> 37990488
> >> 19996 37992380 37992384 37992384 37992384 37992384 37992384 37992388
> >> 37992388
> >> 19997 37994280 37994280 37994284 37994284 37994284 37994288 37994288
> >> 37994288
> >> 19998 37996180 37996184 37996184 37996184 37996184 37996184 37996188
> >> 37996188
> >> 19999 37998080 37998080 37998084 37998084 37998084 37998088 37998088
> >> 37998088
> >> 20000 37999980 37999984 37999984 37999984 37999984 37999984 37999988
> >> 37999988
> >>           1889     1890     1891     1892     1893     1894     1895
> >> 1896
> >> 19991 37982888 37982888 37982892 37982892 37982892 37982896 37982896
> >> 37982896
> >> 19992 37984788 37984792 37984792 37984792 37984792 37984792 37984796
> >> 37984796
> >> 19993 37986688 37986688 37986692 37986692 37986692 37986696 37986696
> >> 37986696
> >> 19994 37988588 37988592 37988592 37988592 37988592 37988592 37988596
> >> 37988596
> >> 19995 37990488 37990488 37990492 37990492 37990492 37990496 37990496
> >> 37990496
> >> 19996 37992388 37992392 37992392 37992392 37992392 37992392 37992396
> >> 37992396
> >> 19997 37994288 37994288 37994292 37994292 37994292 37994296 37994296
> >> 37994296
> >> 19998 37996188 37996192 37996192 37996192 37996192 37996192 37996196
> >> 37996196
> >> 19999 37998088 37998088 37998092 37998092 37998092 37998096 37998096
> >> 37998096
> >> 20000 37999988 37999992 37999992 37999992 37999992 37999992 37999996
> >> 37999996
> >>           1897     1898     1899     1900
> >> 19991 37982896 37982896 37982900 37982900
> >> 19992 37984796 37984800 37984800 37984800
> >> 19993 37986696 37986696 37986700 37986700
> >> 19994 37988596 37988600 37988600 37988600
> >> 19995 37990496 37990496 37990500 37990500
> >> 19996 37992396 37992400 37992400 37992400
> >> 19997 37994296 37994296 37994300 37994300
> >> 19998 37996196 37996200 37996200 37996200
> >> 19999 37998096 37998096 37998100 37998100
> >> 20000 37999996 38000000 38000000 38000000
> >> > tail(out2)
> >>           1881     1882     1883     1884     1885     1886     1887
> >> 1888
> >> 19991 37982881 37982882 37982883 37982884 37982885 37982886 37982887
> >> 37982888
> >> 19992 37984781 37984782 37984783 37984784 37984785 37984786 37984787
> >> 37984788
> >> 19993 37986681 37986682 37986683 37986684 37986685 37986686 37986687
> >> 37986688
> >> 19994 37988581 37988582 37988583 37988584 37988585 37988586 37988587
> >> 37988588
> >> 19995 37990481 37990482 37990483 37990484 37990485 37990486 37990487
> >> 37990488
> >> 19996 37992381 37992382 37992383 37992384 37992385 37992386 37992387
> >> 37992388
> >> 19997 37994281 37994282 37994283 37994284 37994285 37994286 37994287
> >> 37994288
> >> 19998 37996181 37996182 37996183 37996184 37996185 37996186 37996187
> >> 37996188
> >> 19999 37998081 37998082 37998083 37998084 37998085 37998086 37998087
> >> 37998088
> >> 20000 37999981 37999982 37999983 37999984 37999985 37999986 37999987
> >> 37999988
> >>           1889     1890     1891     1892     1893     1894     1895
> >> 1896
> >> 19991 37982889 37982890 37982891 37982892 37982893 37982894 37982895
> >> 37982896
> >> 19992 37984789 37984790 37984791 37984792 37984793 37984794 37984795
> >> 37984796
> >> 19993 37986689 37986690 37986691 37986692 37986693 37986694 37986695
> >> 37986696
> >> 19994 37988589 37988590 37988591 37988592 37988593 37988594 37988595
> >> 37988596
> >> 19995 37990489 37990490 37990491 37990492 37990493 37990494 37990495
> >> 37990496
> >> 19996 37992389 37992390 37992391 37992392 37992393 37992394 37992395
> >> 37992396
> >> 19997 37994289 37994290 37994291 37994292 37994293 37994294 37994295
> >> 37994296
> >> 19998 37996189 37996190 37996191 37996192 37996193 37996194 37996195
> >> 37996196
> >> 19999 37998089 37998090 37998091 37998092 37998093 37998094 37998095
> >> 37998096
> >> 20000 37999989 37999990 37999991 37999992 37999993 37999994 37999995
> >> 37999996
> >>           1897     1898     1899     1900
> >> 19991 37982897 37982898 37982899 37982900
> >> 19992 37984797 37984798 37984799 37984800
> >> 19993 37986697 37986698 37986699 37986700
> >> 19994 37988597 37988598 37988599 37988600
> >> 19995 37990497 37990498 37990499 37990500
> >> 19996 37992397 37992398 37992399 37992400
> >> 19997 37994297 37994298 37994299 37994300
> >> 19998 37996197 37996198 37996199 37996200
> >> 19999 37998097 37998098 37998099 37998100
> >> 20000 37999997 37999998 37999999 38000000
> >> > tail(res)
> >>       1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893
> 1894
> >> 19991   -1   -2    1    0   -1    2    1    0   -1   -2    1    0   -1
>    2
> >> 19992   -1    2    1    0   -1   -2    1    0   -1    2    1    0   -1
>   -2
> >> 19993   -1   -2    1    0   -1    2    1    0   -1   -2    1    0   -1
>    2
> >> 19994   -1    2    1    0   -1   -2    1    0   -1    2    1    0   -1
>   -2
> >> 19995   -1   -2    1    0   -1    2    1    0   -1   -2    1    0   -1
>    2
> >> 19996   -1    2    1    0   -1   -2    1    0   -1    2    1    0   -1
>   -2
> >> 19997   -1   -2    1    0   -1    2    1    0   -1   -2    1    0   -1
>    2
> >> 19998   -1    2    1    0   -1   -2    1    0   -1    2    1    0   -1
>   -2
> >> 19999   -1   -2    1    0   -1    2    1    0   -1   -2    1    0   -1
>    2
> >> 20000   -1    2    1    0   -1   -2    1    0   -1    2    1    0   -1
>   -2
> >>       1895 1896 1897 1898 1899 1900
> >> 19991    1    0   -1   -2    1    0
> >> 19992    1    0   -1    2    1    0
> >> 19993    1    0   -1   -2    1    0
> >> 19994    1    0   -1    2    1    0
> >> 19995    1    0   -1   -2    1    0
> >> 19996    1    0   -1    2    1    0
> >> 19997    1    0   -1   -2    1    0
> >> 19998    1    0   -1    2    1    0
> >> 19999    1    0   -1   -2    1    0
> >> 20000    1    0   -1    2    1    0
> >> >
> >> > res[8831,]
> >>    [1]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>   [25]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>   [49]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>   [73]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>   [97]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>  [121]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>  [145]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>  [169]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>  [193]  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
> >>  0  0
> >>  [217] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [241] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [265] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [289] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [313] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [337] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [361] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [385] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [409] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>  [433] -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0  1  0 -1  0
> >>  1  0
> >>
> >>
> >> >>> Ted Rosenbaum  04/30/13 3:18 PM >>>
> >> Dear Matteo,
> >>
> >> Thanks for your response.
> >>
> >> You approach would work except that my computer can't handle it well
> since
> >> it loads everything into memory.  That is why I used the more
> complicated
> >> approach outlined here (
> >> http://cran.r-project.org/web/packages/raster/vignettes/functions.pdf).
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Apr 30, 2013 at 3:24 AM, Matteo Mattiuzzi <
> >> matteo.mattiuzzi@boku.ac.at> wrote:
> >>
> >> > Dear Ted,
> >> >
> >> >
> >> > Isn't it simply that?
> >> > out <- raster(nrow=20000,ncol=1900)#modisRasterAll
> >> > out[] <- 1:ncell(out)
> >> >
> >> >
> >> > tail(out)
> >> >
> >> >
> >> >
> >> >
> >> > >>> Ted Rosenbaum  04/29/13 11:28 PM >>>
> >> > Hi,
> >> >
> >> > I am trying to assign a unique ID number to each tile in a large
> raster
> >> > file.
> >> >
> >> > My code is below:
> >> >
> >> > ##Write tile indexed MODIS file
> >> > #Prelims
> >> > nColVal<-190
> >> > outFileName <- 'tmp.grd'
> >> >
> >> > #Set up raster
> >> > out <- raster(nrow=20000,ncol=nColVal)#modisRasterAll
> >> >
> >> > ##Get number of colums and blocksize
> >> > outCol <- ncol(out)
> >> > bs <- blockSize(out)
> >> >
> >> > ##Write to file
> >> > out <- writeStart(out,outFileName,overwrite=TRUE)
> >> > for(i in 1:bs$n){
> >> >   print(i)
> >> >   out <-
> >> >
> >> >
> >>
> writeValues(out,((outCol*(bs$row[i]-1))+1):((bs$row[i]+bs$nrows[i]-1)*outCol),bs$row[i])
> >> > }
> >> > out <- writeStop(out)
> >> >
> >> > tail(out)
> >> >
> >> > When I run this code with nColVal=190, I get the expected result from
> >> > tail(out): a sequence of numbers.  However, when I run this code with
> >> > nColVal=1900, I do not and instead get
> >> > "37999996 38000000 38000000 38000000" for out[20000,1897:1900].
> >> >
> >> > I would like to get instead "37999997 37999998 37999999 38000000".
> >> >
> >> > Thanks for any assistance,
> >> >
> >> > Ted Rosenbaum
> >> > Department of Economics
> >> > Yale University
> >> >
> >> >     [[alternative HTML version deleted]]
> >> >
> >> > _______________________________________________
> >> > R-sig-Geo mailing list
> >> > R-sig-Geo@r-project.org
> >> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >> >
> >> >
> >> >
> >>
> >> _______________________________________________
> >> R-sig-Geo mailing list
> >> R-sig-Geo@r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>
> >
> >
>

	[[alternative HTML version deleted]]

