[R] How to save very large matrix?

[Rui Barradas]

Hello,

You can use the argument to write.csv or write.table  append = TRUE to 
write the matrix in chunks. Something like the following.



bigwrite <- function(x, file, rows = 1000L, ...){
	passes <- NROW(x) %/% rows
	remaining <- NROW(x) %% rows
	k <- 1L
	write.table(x[k:rows, ], file, row.names = FALSE, ...)
	k <- k + rows
	for(i in seq_len(passes)[-1]){
		write.table(x[k:(rows*i), ], file, append = TRUE, row.names = FALSE, 
col.names = FALSE, ...)
		k <- k + rows
	}
	if(remaining > 0)
		write.table(x[k:NROW(x), ], file, append = TRUE, row.names = FALSE, 
col.names = FALSE, ...)
}

f <- "temp"
m <- matrix(0, 50012, 10)

bigwrite(m, f, sep = ",")  # Use 'sep' to get a csv file



Hope this helps,

Rui Barradas


Em 29-10-2013 19:27, Petar Milin escreveu:
> Hello!
> I have a very large matrix of results: 50000x100000. I saved it as RDS, but I would also need to save it as txt or csv. Is there a way to do it? Now, with write.table I am receiving an error:
> Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol,  :
>    long vectors not supported yet: io.c:1116
>
> Please, help! Many thanks!
>
> PM
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>