Mark:
   
  Here is the information I didn't provide in my earlier post. R version is R2.2.1 running on Windows XP.  My dataset has 16 variables with following data type.
  ColNumber:   1              2              3  .......16
  Datatypes:
    "numeric","numeric","numeric","numeric","numeric","numeric","character","numeric","numeric","character","character","numeric","numeric","numeric","numeric","numeric","numeric","numeric"
   
  Variable (2) which is numeric and variables denoted as character are to be treated as dummy variables in the regression. 
   
  Search in R help list  suggested I can use read.csv with colClasses option also instead of using scan() and then converting it to dataframe as you suggested. I am trying both these methods but unable to resolve syntactical error. 
   
  >coltypes<- c("numeric","factor","numeric","numeric","numeric","numeric","factor","numeric","numeric","factor","factor","numeric","numeric","numeric","numeric","numeric","numeric","numeric")
   
  >mydf <- read.csv("C:/temp/data.csv", header=FALSE, colClasses = coltypes, strip.white=TRUE)
   
  ERROR: Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  : 
        scan() expected 'a real', got 'V1'
   
  No idea whats the problem.
   
  AS PER YOUR SUGGESTION I TRIED scan() as follows:
   
  >coltypes<-c("numeric","factor","numeric","numeric","numeric","numeric","factor","numeric","numeric","factor","factor","numeric","numeric","numeric","numeric","numeric","numeric","numeric")
  >x<-scan(file = "C:/temp/data.dbf",what=as.list(coltypes),sep=",",quiet=TRUE,skip=1) 
  >names(x)<-scan(file = "C:/temp/data.dbf",what="",nlines=1, sep=",") 
  >x<-as.data.frame(x) 
   
  This is working fine but x has no data in it and contains
  > x
   
   [1] X._.   NA.    NA..1  NA..2  NA..3  NA..4  NA..5  NA..6  NA..7  NA..8  NA..9  NA..10 NA..11
[14] NA..12 NA..13 NA..14 NA..15 NA..16
<0 rows> (or 0-length row.names)
   
  Please let me know how to properly use scan or colClasses option.
   
  Sachin

   
   
  

Mark Stephens <markjs1@googlemail.com> wrote:
  Sachin,
With your dummies stored as integer, the size of your object would appear
to be 350000 * (4*250 + 8*16) bytes = 376MB.
You said "PC" but did not provide R version information, assuming windows
then ...
With 1GB RAM you should be able to load a 376MB object into memory. If you
can store the dummies as 'raw' then object size is only 126MB.
You don't say how you attempted to load the data. Assuming your input data
is in text file (or can be) have you tried scan()? Setup the 'what' argument
with length 266 and make sure the dummy column are set to integer() or
raw(). Then x = scan(...); class(x)=" data.frame".
What is the result of memory.limit()? If it is 256MB or 512MB, then try
starting R with --max-mem-size=800M (I forget the syntax exactly). Leave a
bit of room below 1GB. Once the object is in memory R may need to copy it
once, or a few times. You may need to close all other apps in memory, or
send them to swap.
I don't really see why your data should not fit into the memory you have.
Purchasing an extra 1GB may help. Knowing the object size calculation (as
above) should help you guage whether it is worth it.
Have you used process monitor to see the memory growing as R loads the
data? This can be useful.
If all the above fails, then consider 64-bit and purchasing as much memory
as you can afford. R can use over 64GB RAM+ on 64bit machines. Maybe you can
hire some time on a 64-bit server farm - i heard its quite cheap but never
tried it myself. You shouldn't need to go that far with this data set
though.
Hope this helps,
Mark


Hi Roger,

I want to carry out regression analysis on this dataset. So I believe I
can't read the dataset in chunks. Any other solution?

TIA
Sachin


roger koenker < rkoenker@uiuc.edu> wrote:
You can read chunks of it at a time and store it in sparse matrix
form using the packages SparseM or Matrix, but then you need
to think about what you want to do with it.... least squares sorts
of things are ok, but other options are somewhat limited...


url: www.econ.uiuc.edu/~roger Roger Koenker
email rkoenker@uiuc.edu Department of Economics
vox: 217-333-4558 University of Illinois
fax: 217-244-6678 Champaign, IL 61820


On Apr 24, 2006, at 12:41 PM, Sachin J wrote:

> Hi,
>
> I have a dataset consisting of 350,000 rows and 266 columns. Out
> of 266 columns 250 are dummy variable columns. I am trying to read
> this data set into R dataframe object but unable to do it due to
> memory size limitations (object size created is too large to handle
> in R). Is there a way to handle such a large dataset in R.
>
> My PC has 1GB of RAM, and 55 GB harddisk space running windows XP.
>
> Any pointers would be of great help.
>
> TIA
> Sachin
>

[[alternative HTML version deleted]]

______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html


		
---------------------------------

	[[alternative HTML version deleted]]