[R] sequential processing

[bogdan romocea]

One option for processing very large files with R is split:
  ## split a large file into pieces
  #--parameters: the folder, file and number of parts
  FLD=/home/user/data
  F=very_large_file.dat
  parts=50
  #---split
  cd $FLD
  fn=`echo $F | awk -F\. '{print $1}'`  #file name without extension
  ln=`wc -l $F | awk '{print $1}'`  #number of lines in the file
  forsplit=`expr $ln / $parts + 1`  #number of lines in each part
  echo "====== $F will be split in $parts parts of $forsplit lines each."
  split -l $forsplit $F $fn
You could also load the entire file into a DBMS then pull parts of it
into R, or read specific lines through a pipe e.g.
readLines(pipe("sed, grep, python... command")).

Don't try to replicate the SAS processing into R. The exact
translations of the SAS DATA STEP usage of _N_, first., last., retain
etc into R would be: inefficient, ugly, retrogressive, wrong, rigid,
complicated, silly and so on. For a start, read up on indexing - this
seemingly simple and innocuous R feature is in fact far more powerful
than the entire DATA STEP with its whole bag of tricks. Then search
the list for similar questions, for example
http://thread.gmane.org/gmane.comp.lang.r.general/44332/focus=44343


> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Gerard Smits
> Sent: Sunday, January 21, 2007 2:22 PM
> To: r-help at stat.math.ethz.ch
> Subject: [R] sequential processing
>
> Like many others, I am new to R but old to SAS.
>
> Am I correct in understanding that R processes a data frame in a
> sequential ly?  This would imply that large input files could be
> read, without the need to load the entire file into memory.
> Related to the manner of reading a frame, I have been looking for the
> equivalent of SAS _n_ (I realize that I can use a variant of which to
> identify an index value) as well as  useful SAS features such as
> first., last., retain, etc.  Any help with this conversion
> appreciated.
>
> Thanks,
>
> Gerard Smits
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>