[R] Exporting large datasets

[DrDimo]

Thanks guys, writing smaller pieces did the trick. Still takes about 4 hours
to do but that more manageable than over 30 hours without even writing a
line.


jholtman wrote:
> 
> What do you want to export it to and in what format?  You will probably
> have
> a problem trying to write that large an object out at one time.  Try
> instead
> opening a connecting and the writing much smaller pieces to the file:
> 
> x <- matrix(runif(10000), 100)
> x[,1] <- 1:100  # put some labels for testing
> x.out <- file('/test.csv', 'w')
> # not efficient, just for demonstration
> for (i in 1:100) write.table(x[i,, drop=FALSE], file=x.out,
>     sep=',', col.names=FALSE, row.names=FALSE)
> close(x.out)
> 
> 
> 
> On Mon, Jun 29, 2009 at 2:51 AM, DrDimo
> <dmitri.mouradov at ludwig.edu.au>wrote:
> 
>>
>> Hi all,
>>
>> I am currently working on a particularly large (data wise) project, the
>> problem I am having is with exporting the data (for use later on). My
>> current sample dataset contains about 400million entries (approximately
>> 20000 columns  x 20000 rows). I tried using the write.table command, but
>> 8
>> hours later its still going. I will be trying write.matrix (MASS) next as
>> it
>> is supposed to handle things a bit quicker, but I thought I would ask if
>> anyone has had experience with exporting large datasets and what is the
>> quickest method to do so.
>>
>> Thanks in advance
>>
>> -Dmitri
>> --
>> View this message in context:
>> http://www.nabble.com/Exporting-large-datasets-tp24249149p24249149.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
>> and provide commented, minimal, self-contained, reproducible code.
>>
> 
> 
> 
> -- 
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
> 
> What is the problem that you are trying to solve?
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
> 

-- 
View this message in context: http://www.nabble.com/Exporting-large-datasets-tp24249149p24299303.html
Sent from the R help mailing list archive at Nabble.com.