Thank you so much Steve.

The computer I'm currently working with is a 32 bit windows 7 OS. And RAM
is only 4GB so I guess thats a big limitation.
El 18/08/2013 03:11, "Steve Lianoglou" <lianoglou.steve@gene.com> escribió:

> Hi Paul,
>
> On Sun, Aug 18, 2013 at 12:56 AM, Paul Bernal <paulbernal07@gmail.com>
> wrote:
> > Thanks a lot for the valuable information.
> >
> > Now my question would necessarily be, how many columns can R handle,
> > provided that I have millions of rows and, in general, whats the maximum
> > amount of rows and columns that R can effortlessly handle?
>
> This is all determined by your RAM.
>
> Prior to R-3.0, R could only handle vectors of length 2^31 - 1. If you
> were working with a matrix, that meant that you could only have that
> many elements in the entire matrix.
>
> If you were working with a data.frame, you could have data.frames with
> 2^31-1 rows, and I guess as many columns, since data.frames are really
> a list of vectors, the entire thing doesn't have to be in one
> contiguous block (and addressable that way)
>
> R-3.0 introduced "Long Vectors" (search for that section in the release
> notes):
>
> https://stat.ethz.ch/pipermail/r-announce/2013/000561.html
>
> It almost doubles the size of a vector that R can handle (assuming you
> are running 64bit). So, if you've got the RAM, you can have a
> data.frame/data.table w/ billion(s) of rows, in theory.
>
> To figure out how much data you can handle on your machine, you need
> to know the size of real/integer/whatever and the number of elements
> of those you will have so you can calculate the amount of RAM you need
> to load it all up.
>
> Lastly, I should mention there are packages that let you work with
> "out of memory" data, like bigmemory, biglm, ff. Look at the HPC Task
> view for more info along those lines:
>
> http://cran.r-project.org/web/views/HighPerformanceComputing.html
>
>
> >
> > Best regards and again thank you for the help,
> >
> > Paul
> > El 18/08/2013 02:35, "Steve Lianoglou" <lianoglou.steve@gene.com>
> escribió:
> >
> >> Hi Paul,
> >>
> >> First: please keep your replies on list (use reply-all when replying
> >> to R-help lists) so that others can help but also the lists can be
> >> used as a resource for others.
> >>
> >> Now:
> >>
> >> On Aug 18, 2013, at 12:20 AM, Paul Bernal <paulbernal07@gmail.com>
> wrote:
> >>
> >> > Can R really handle millions of rows of data?
> >>
> >> Yup.
> >>
> >> > I thought it was not possible.
> >>
> >> Surprise :-)
> >>
> >> As I type, I'm working with a ~5.5 million row data.table pretty
> >> effortlessly.
> >>
> >> Columns matter too, of course -- RAM is RAM, after all and you've got
> >> to be able to fit the whole thing into it if you want to use
> >> data.table. Once loaded, though, data.table enables one to do
> >> split/apply/combine calculations over these data quite efficiently.
> >> The first time I used it, I was honestly blown away.
> >>
> >> If you find yourself wanting to work with such data, you could do
> >> worse than read through data.table's vignette and FAQ and give it a
> >> spin.
> >>
> >> HTH,
> >>
> >> -steve
> >>
> >> --
> >> Steve Lianoglou
> >> Computational Biologist
> >> Bioinformatics and Computational Biology
> >> Genentech
> >>
> >
> >         [[alternative HTML version deleted]]
> >
> >
> > ______________________________________________
> > R-help@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
>
> --
> Steve Lianoglou
> Computational Biologist
> Bioinformatics and Computational Biology
> Genentech
>

	[[alternative HTML version deleted]]

