[R] read multiple large files into one dataframe

[Simon Pickett]

can you provide reproducible code please?

even a fake example would help.

I would

1) set up a loop to read in each file from a directory
2)  inside the loop chop up/ aggregate the data, each file in turn and spit 
each new aggreagated file out to a directory using write.table(). This will 
reduce the memory needed by only including the info you want. Make sure each 
file is a data frame with the same names.
3) set up a new loop to read in each new small file and rbind them all 
together to make your new "master file".

The R gurus may have a more parsimonious solution.

HTH

Simon.


----- Original Message ----- 
From: "SYKES, Jennifer" <Jennifer.SYKES at nats.co.uk>
To: <r-help at r-project.org>
Sent: Wednesday, May 13, 2009 11:45 AM
Subject: [R] read multiple large files into one dataframe


> Hello
>
>
>
> Apologies if this is a simple question, I have searched the help and
> have not managed to work out a solution.
>
> Does anybody know an efficient method for reading many text files of the
> same format into one table/dataframe?
>
>
>
> I have around 90 files that contain continuous data over 3 months but
> that are split into individual days data and I need the whole 3 months
> in one file for analysis.  Each days file contains a large amount of
> data (approx 30MB each) and so I need a memory efficient method to merge
> all of the files into the one dataframe object.  From what I have read I
> will probably want to avoid using for loops etc?  All files are in the
> same directory, none have a header row, and each contain around 180,000
> rows and the same 25 columns/variables.  Any suggested packages/routines
> would be very useful.
>
>
>
> Thanks
>
>
>
> Jennifer
>
>
>
>
>
>
>
> -----------------------------------------
> *******************************************************************If
> you are not the intended recipient, please notify our Help Desk at
> Email postmaster at nats.co.uk immediately. You should not copy or use
> this email or attachment(s) for any purpose nor disclose their
> contents to any other person. NATS computer systems may be
> monitored and communications carried on them recorded, to secure
> the effective operation of the system and for other lawful
> purposes. Please note that neither NATS nor the sender accepts any
> responsibility for viruses or any losses caused as a result of
> viruses and it is your responsibility to scan or otherwise check
> this email and any attachments. NATS means NATS (En Route) plc
> (company number: 4129273), NATS (Services) Ltd (company number
> 4129270), NATSNAV Ltd (company number: 4164590) or NATS Ltd
> (company number 3155567) or NATS Holdings Ltd (company number
> 4138218). All companies are registered in England and their
> registered office is at 5th Floor, Brettenham House South,
> Lancaster Place, London, WC2E 7EN.
> **********************************************************************
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>