[Rd] truncation/rounding bug with write.csv

Wed Mar 14 17:02:41 CET 2018

I ran this code in RStudio Server on a linux machine, but I don’t know the version offhand.  I will try to get it tomorrow.  Thanks.

Thanks,
Greg Michaelson
www.datarobot.com
704-981-1118

> On Mar 14, 2018, at 4:47 PM, Joris Meys <jorismeys at gmail.com> wrote:
> 
> To my surprise, I can confirm on Windows 10 using R 3.4.3 . As tail is not recognized by Windows cmd, I replaced with:
> 
> system('powershell -nologo "& "Get-Content -Path temp.csv -Tail 1')
> 
> The last line shows only 7 digits after the decimal, whereas the first have 15 digits after the decimal. I agree with Dirk though, 1.6Gb csv files are not the best way to work with datasets.
> 
> Cheers
> Joris
> 
> 
> 
> On Wed, Mar 14, 2018 at 1:53 PM, Dirk Eddelbuettel <edd at debian.org <mailto:edd at debian.org>> wrote:
> 
> What OS are you on?  On Ubuntu 17.10 with R 3.4.3 all seems well (see
> below for your example, I just added a setwd()).
> 
> [ That said, I long held a (apparently minority) view that csv is for all
> intends and purposes a less-than-ideal format.  If you have that much data,
> you do generally not want to serialize it back and forth as that is slow, and
> may drop precision.  The rds format is great for R alone; we now have C code
> to read it from other apps (in the librdata repo by Evan Miller).  Different
> portable serializations work too (protocol buffer, msgpack, ...), there are
> databases and on and on... ]
> 
> Dirk
> 
> 
> R> df <- data.frame(replicate(100, runif(1000000, 0,1)))
> R> setwd("/tmp")
> R> write.csv(df, "temp.csv")
> R> system('tail -n1 temp.csv')
> "1000000",0.11496100993827,0.740764639340341,0.519190795486793,0.736045523779467,0.537115448853001,0.769496953347698,0.102257401449606,0.437617724528536,0.173321532085538,0.351960731903091,0.397348914295435,0.496789071243256,0.463006566744298,0.573105450021103,0.575196429155767,0.821617329493165,0.112913676071912,0.187580146361142,0.121353451395407,0.576333721866831,0.00763232703320682,0.468676633667201,0.451408475637436,0.0172415724955499,0.946199159137905,0.439950440311804,0.109224532730877,0.657066411571577,0.0524766123853624,0.54859598656185,0.94473168021068,0.500153199071065,0.636756601976231,0.221365773351863,0.620196332456544,0.559639401268214,0.198483835440129,0.397874651942402,0.710652963491157,0.317212327616289,0.239299293374643,0.0606942125596106,0.165786643279716,0.667431530542672,0.436631754040718,0.812185280025005,0.374252707697451,0.421187321422622,0.730321826180443,0.904493971262127,0.399387824581936,0.650714065413922,0.594219180056825,0.147960299625993,0.941945064114407,0.357223904458806,0.275038427906111,0.191008436959237,0.957893384154886,0.211530723143369,0.680650093592703,0.503884038887918,0.754094189498574,0.74776051659137,0.673691919771954,0.236221367260441,0.825558929471299,0.21071959589608,0.246618688805029,0.686810691142455,0.0247942050918937,0.572868114337325,0.494058627169579,0.684360746992752,0.0139967589639127,0.626861660508439,0.417218193877488,0.410173830809072,0.390906651504338,0.477168896235526,0.382211019750684,0.597674581920728,0.198329919017851,0.0684413285925984,0.450342149706557,0.133007253985852,0.755873151356354,0.372862737858668,0.762442974606529,0.582133987685665,0.692048883531243,0.259269661735743,0.147847984684631,0.635266482364386,0.320955650880933,0.00151186063885689,0.446474697208032,0.0673662247136235,0.791947861900553,0.0973296447191387
> R> system('head -n2 temp.csv')
> "","X1","X2","X3","X4","X5","X6","X7","X8","X9","X10","X11","X12","X13","X14","X15","X16","X17","X18","X19","X20","X21","X22","X23","X24","X25","X26","X27","X28","X29","X30","X31","X32","X33","X34","X35","X36","X37","X38","X39","X40","X41","X42","X43","X44","X45","X46","X47","X48","X49","X50","X51","X52","X53","X54","X55","X56","X57","X58","X59","X60","X61","X62","X63","X64","X65","X66","X67","X68","X69","X70","X71","X72","X73","X74","X75","X76","X77","X78","X79","X80","X81","X82","X83","X84","X85","X86","X87","X88","X89","X90","X91","X92","X93","X94","X95","X96","X97","X98","X99","X100"
> "1",0.995067856274545,0.0237177284434438,0.839840568602085,0.99880409357138,0.455015312181786,0.967688028467819,0.191194181796163,0.903533136472106,0.570170691236854,0.86230118968524,0.23530788696371,0.30707904486917,0.256274404237047,0.369592409580946,0.989929250674322,0.50812312704511,0.806819133926183,0.536566868191585,0.0863138805143535,0.294523851014674,0.676951135974377,0.195627561537549,0.261776751372963,0.383222601376474,0.578275503357872,0.79082652577199,0.19860127940774,0.0204593606758863,0.659964868798852,0.42379029514268,0.69516694964841,0.0594558380544186,0.124592808773741,0.289328144863248,0.524508266709745,0.84306427766569,0.317027662880719,0.273440480465069,0.111866136547178,0.217484838794917,0.354757327819243,0.973936082562432,0.673076402861625,0.300948366522789,0.219195493729785,0.912278874544427,0.276768424082547,0.959344451315701,0.500720858341083,0.431024399353191,0.814444699790329,0.0738761406391859,0.600137831410393,0.639816240407526,0.405302967177704,0.941259450744838,0.190415472723544,0.0382565588224679,0.486769351176918,0.127647049957886,0.558708024444059,0.686994878342375,0.176803215174004,0.794697789475322,0.59406904829666,0.0897431457415223,0.196549082174897,0.0750515828840435,0.736311340238899,0.00494878669269383,0.383522965712473,0.960385771468282,0.101023471681401,0.209177070530131,0.798869548132643,0.147874428424984,0.187238642480224,0.148522146046162,0.32379064662382,0.620601811446249,0.201180462958291,0.179565666476265,0.466121524339542,0.245493365218863,0.980698639061302,0.342919659335166,0.387780519668013,0.393966492731124,0.148554262006655,0.521724705817178,0.722740866011009,0.105151653522626,0.461909410310909,0.905382365221158,0.0736293855588883,0.636923864483833,0.540197744267061,0.425208077067509,0.666353516280651,0.584139186656103
> R>
> 
> --
> http://dirk.eddelbuettel.com <http://dirk.eddelbuettel.com/> | @eddelbuettel | edd at debian.org <mailto:edd at debian.org>
> ______________________________________________
> R-devel at r-project.org <mailto:R-devel at r-project.org> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel <https://stat.ethz.ch/mailman/listinfo/r-devel>
> 
> 
> 
> -- 
> Joris Meys
> Statistical consultant
> 
> Department of Data Analysis and Mathematical Modelling
> Ghent University
> Coupure Links 653, B-9000 Gent (Belgium)
>  <https://maps.google.com/?q=Coupure+links+653,%C2%A0B-9000+Gent,%C2%A0Belgium&entry=gmail&source=g>
> 
> -----------
> Biowiskundedagen 2017-2018
> http://www.biowiskundedagen.ugent.be/ <http://www.biowiskundedagen.ugent.be/>
> 
> -------------------------------
> Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php <http://helpdesk.ugent.be/e-maildisclaimer.php>

	[[alternative HTML version deleted]]