[R] Failing on reading a "slightly big" dataset

David Khabie-Zeitoune dave at evocapital.com
Mon Jul 5 12:26:30 CEST 2004


Try specifying quote=NULL as an argument to read.table. It could be that
one of your fields has a quote symbol in it.

-----Original Message-----
From: Ajay Shah [mailto:ajayshah at mayin.org] 
Sent: 05 July 2004 11:15
To: r-help
Subject: [R] Failing on reading a "slightly big" dataset


I have a file with 4 columns per line, all pipe delimited.

$ wc -l cmie_firm_data.text 
89325 cmie_firm_data.text
$ ls -al cmie_firm_data.text 
-rw-r--r--    1 ajayshah ajayshah  4415637 Jul  5 15:25
cmie_firm_data.text
$ awk -F\| '(NF != 4)' cmie_firm_data.text 
$ head cmie_firm_data.text 
All figures are for the year 20030331|||
Company|GVA Less Interest (Rs. thousand)|Interest (Rs. thousand)|GVA 
Company|(Rs. thousand)
'R' INVEST PVT. LTD.|-510.45|0.18|-510.27
20 MICRONS LTD.|60700|41200|101900
20TH CENTURY FOX CORPN. (INDIA) PVT. LTD.|50|0.33|50.33
21ST CENTURY AUTOMOTIVE INDIA LTD.|201.14|0.19|201.33
21ST CENTURY ENTERTAINMENT PVT. LTD.|-6.10|0|-6.10
21ST CENTURY EQUIPMENTS PVT. LTD.|-1599.53|1262.76|-336.77
21ST CENTURY INFRASTRUCTURE (INDIA) PVT. LTD.|140.48|1.74|142.22 21ST
CENTURY PEST CONTROL SERVICES LTD.|50.21|7.13|57.34

When I try to read this into R, I get a mysterious error, and then it
reads only 38,244 observations. Any idea what might be going wrong? In
case it matters, I'm confident this is a Unix text file; no DOSisms of
CR-LF here.

$ R --vanilla < picture.R 

R : Copyright 2004, The R Foundation for Statistical Computing Version
1.9.1  (2004-06-21), ISBN 3-900051-00-3

> firms <- read.table("cmie_firm_data.text", sep="|", skip=2,
+                     col.names=c("name", "gva.less.interest",
+                       "interest", "gva"))
Warning message: 
number of items read is not a multiple of the number of columns 

> summary(firms)
                                            name       gva.less.interest

 20 MICRONS LTD.                              :    1   Min.
:-1.049e+07  
 20TH CENTURY FOX CORPN. (INDIA) PVT. LTD.    :    1   1st Qu.:
1.720e+01  
 21ST CENTURY AUTOMOTIVE INDIA LTD.           :    1   Median :
5.664e+02  
 21ST CENTURY ENTERTAINMENT PVT. LTD.         :    1   Mean   :
4.858e+04  
 21ST CENTURY EQUIPMENTS PVT. LTD.            :    1   3rd Qu.:
2.587e+03  
 21ST CENTURY INFRASTRUCTURE (INDIA) PVT. LTD.:    1   Max.   :
1.968e+08  
 (Other)                                      :38244   NA's   :
1.000e+00  
    interest              gva            
 Min.   :0.000e+00   Min.   :-2.349e+06  
 1st Qu.:5.500e-01   1st Qu.: 4.909e+01  
 Median :4.462e+01   Median : 7.711e+02  
 Mean   :6.301e+03   Mean   : 4.565e+04  
 3rd Qu.:5.788e+02   3rd Qu.: 3.436e+03  
 Max.   :9.558e+06   Max.   : 2.004e+08  
 NA's   :2.530e+02   NA's   : 2.530e+02  

Sniffing, I find that the last observation in the data frame `firms' is
wonky:

> print(firms[38249,])
                          name gva.less.interest interest     gva
38249 YOUNG POLYMERS PVT. LTD.           2542.08   652.71 3194.79
> system("grep -n '^YOUNG POLY' cmie*")
88904:YOUNG POLYMERS PVT. LTD.|2542.08|652.71|3194.79

where we see that YOUNG POLYMERS was observation 88,904 in the file. How
did it become #38249 to R?

And,

> print(firms[38250,])

makes him go nuts --

38250 YOUNG WOMENS CHRISTIAN ASSN. OF
INDIA|9477.71|24.82|9502.53\nYOUNGMAN WOOL LEN MILLS
LTD.|5395.08|6316.75|11711.83\nYOUNGSTAR CONSTRUCTION PVT. LTD.|850.71
|128.07|978.78\nYOUR INVESTMENT (INDIA) LTD.|90.85|0|90.85\nYOURCHOICE 
|CHIT FUND
 PVT. LTD.|0|0|0\nYOUTH FORUM TOWERS & CONSTRUCTION PVT.
LTD.|289.79|1.75|291.54 \nYOUTH PROMOTERS PVT.
LTD.|104.87|30.23|135.10\nYU BO INVST. CO. PVT. LTD.|708.
08|5209.60|5917.68\nYU TECHNOLOGIES PVT. 
08|LTD.|923.79|46.69|970.48\nYU-MEN TRADEL
INK PVT. LTD.|-321.47|14.35|-307.12\nYUCCA AGENCIES PVT.
LTD.|1243.49|464.33|170 7.82\nYUCON EXPORTS PVT.
LTD.|503.30|326.49|829.79\nYUCON MARKETING & INVSTS. PV T.
LTD.|-4.73|0.20|-4.53\nYUG MARKETING PVT. LTD.|2696.58|304.42|3001\nYUG
TRADE RS PVT. LTD.|-12.25|0.15|-12.10\nYUGAL CHIT FUND & TRADING CO.
PVT. LTD.|-7.52|0 .20|-7.32\nYUGAL KISHORE FABRICS & GARMENTS PVT.
LTD.|914.82|1298.56|2213.38\nYU GANTAR ENGINEERS PVT.
LTD.|193.56|0.69|194.25\nYUGANTAR INVESTMENTS LTD.|44.38|0
.06|44.44\nYUGANTAR TRADING PVT. LTD.|-4.81|0|-4.81\nYUGO INTRACO PVT.
LTD.|1588 .49|29.16|1617.65\nYUGSUDO (INDIA) ENGG. SERVICE

(deleted).

The file is fine:

$ grep -n 'YOUNG WOM' cmie_firm_data.text 
88905:YOUNG WOMEN'S CHRISTIAN ASSN. OF INDIA|9477.71|24.82|9502.53

Any idea what might be going on?

My machine is linux 2.4.17 #2 (got new Debian packages for 1.9.1 today).

-- 
Ajay Shah                                                   Consultant
ajayshah at mayin.org                      Department of Economic Affairs
http://www.mayin.org/ajayshah           Ministry of Finance, New Delhi

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html




More information about the R-help mailing list