[Rd] truncation/rounding bug with write.csv

Wed Mar 14 16:47:45 CET 2018

To my surprise, I can confirm on Windows 10 using R 3.4.3 . As tail is not
recognized by Windows cmd, I replaced with:

system('powershell -nologo "& "Get-Content -Path temp.csv -Tail 1')

The last line shows only 7 digits after the decimal, whereas the first have
15 digits after the decimal. I agree with Dirk though, 1.6Gb csv files are
not the best way to work with datasets.

Cheers
Joris

On Wed, Mar 14, 2018 at 1:53 PM, Dirk Eddelbuettel <edd at debian.org> wrote:

>
> What OS are you on?  On Ubuntu 17.10 with R 3.4.3 all seems well (see
> below for your example, I just added a setwd()).
>
> [ That said, I long held a (apparently minority) view that csv is for all
> intends and purposes a less-than-ideal format.  If you have that much data,
> you do generally not want to serialize it back and forth as that is slow,
> and
> may drop precision.  The rds format is great for R alone; we now have C
> code
> to read it from other apps (in the librdata repo by Evan Miller).
> Different
> portable serializations work too (protocol buffer, msgpack, ...), there are
> databases and on and on... ]
>
> Dirk
>
>
> R> df <- data.frame(replicate(100, runif(1000000, 0,1)))
> R> setwd("/tmp")
> R> write.csv(df, "temp.csv")
> R> system('tail -n1 temp.csv')
> "1000000",0.11496100993827,0.740764639340341,0.519190795486793,0.
> 736045523779467,0.537115448853001,0.769496953347698,0.102257401449606,0.
> 437617724528536,0.173321532085538,0.351960731903091,0.397348914295435,0.
> 496789071243256,0.463006566744298,0.573105450021103,0.575196429155767,0.
> 821617329493165,0.112913676071912,0.187580146361142,0.121353451395407,0.
> 576333721866831,0.00763232703320682,0.468676633667201,0.451408475637436,0.
> 0172415724955499,0.946199159137905,0.439950440311804,0.109224532730877,0.
> 657066411571577,0.0524766123853624,0.54859598656185,0.94473168021068,0.
> 500153199071065,0.636756601976231,0.221365773351863,0.620196332456544,0.
> 559639401268214,0.198483835440129,0.397874651942402,0.710652963491157,0.
> 317212327616289,0.239299293374643,0.0606942125596106,0.165786643279716,0.
> 667431530542672,0.436631754040718,0.812185280025005,0.374252707697451,0.
> 421187321422622,0.730321826180443,0.904493971262127,0.399387824581936,0.
> 650714065413922,0.594219180056825,0.147960299625993,0.941945064114407,0.
> 357223904458806,0.275038427906111,0.191008436959237,0.957893384154886,0.
> 211530723143369,0.680650093592703,0.503884038887918,0.754094189498574,0.
> 74776051659137,0.673691919771954,0.236221367260441,0.825558929471299,0.
> 21071959589608,0.246618688805029,0.686810691142455,0.0247942050918937,0.
> 572868114337325,0.494058627169579,0.684360746992752,0.0139967589639127,0.
> 626861660508439,0.417218193877488,0.410173830809072,0.390906651504338,0.
> 477168896235526,0.382211019750684,0.597674581920728,0.198329919017851,0.
> 0684413285925984,0.450342149706557,0.133007253985852,0.755873151356354,0.
> 372862737858668,0.762442974606529,0.582133987685665,0.692048883531243,0.
> 259269661735743,0.147847984684631,0.635266482364386,0.320955650880933,0.
> 00151186063885689,0.446474697208032,0.0673662247136235,0.
> 791947861900553,0.0973296447191387
> R> system('head -n2 temp.csv')
> "","X1","X2","X3","X4","X5","X6","X7","X8","X9","X10","X11"
> ,"X12","X13","X14","X15","X16","X17","X18","X19","X20","X21"
> ,"X22","X23","X24","X25","X26","X27","X28","X29","X30","X31"
> ,"X32","X33","X34","X35","X36","X37","X38","X39","X40","X41"
> ,"X42","X43","X44","X45","X46","X47","X48","X49","X50","X51"
> ,"X52","X53","X54","X55","X56","X57","X58","X59","X60","X61"
> ,"X62","X63","X64","X65","X66","X67","X68","X69","X70","X71"
> ,"X72","X73","X74","X75","X76","X77","X78","X79","X80","X81"
> ,"X82","X83","X84","X85","X86","X87","X88","X89","X90","X91"
> ,"X92","X93","X94","X95","X96","X97","X98","X99","X100"
> "1",0.995067856274545,0.0237177284434438,0.839840568602085,0.
> 99880409357138,0.455015312181786,0.967688028467819,0.191194181796163,0.
> 903533136472106,0.570170691236854,0.86230118968524,0.23530788696371,0.
> 30707904486917,0.256274404237047,0.369592409580946,0.989929250674322,0.
> 50812312704511,0.806819133926183,0.536566868191585,0.0863138805143535,0.
> 294523851014674,0.676951135974377,0.195627561537549,0.261776751372963,0.
> 383222601376474,0.578275503357872,0.79082652577199,0.19860127940774,0.
> 0204593606758863,0.659964868798852,0.42379029514268,0.69516694964841,0.
> 0594558380544186,0.124592808773741,0.289328144863248,0.524508266709745,0.
> 84306427766569,0.317027662880719,0.273440480465069,0.111866136547178,0.
> 217484838794917,0.354757327819243,0.973936082562432,0.673076402861625,0.
> 300948366522789,0.219195493729785,0.912278874544427,0.276768424082547,0.
> 959344451315701,0.500720858341083,0.431024399353191,0.814444699790329,0.
> 0738761406391859,0.600137831410393,0.639816240407526,0.405302967177704,0.
> 941259450744838,0.190415472723544,0.0382565588224679,0.486769351176918,0.
> 127647049957886,0.558708024444059,0.686994878342375,0.176803215174004,0.
> 794697789475322,0.59406904829666,0.0897431457415223,0.196549082174897,0.
> 0750515828840435,0.736311340238899,0.00494878669269383,0.
> 383522965712473,0.960385771468282,0.101023471681401,0.209177070530131,0.
> 798869548132643,0.147874428424984,0.187238642480224,0.148522146046162,0.
> 32379064662382,0.620601811446249,0.201180462958291,0.179565666476265,0.
> 466121524339542,0.245493365218863,0.980698639061302,0.342919659335166,0.
> 387780519668013,0.393966492731124,0.148554262006655,0.521724705817178,0.
> 722740866011009,0.105151653522626,0.461909410310909,0.905382365221158,0.
> 0736293855588883,0.636923864483833,0.540197744267061,0.425208077067509,0.
> 666353516280651,0.584139186656103
> R>
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Joris Meys
Statistical consultant

Department of Data Analysis and Mathematical Modelling
Ghent University
Coupure Links 653, B-9000 Gent (Belgium)
<https://maps.google.com/?q=Coupure+links+653,%C2%A0B-9000+Gent,%C2%A0Belgium&entry=gmail&source=g>

-----------
Biowiskundedagen 2017-2018
http://www.biowiskundedagen.ugent.be/

-------------------------------
Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php

	[[alternative HTML version deleted]]