[R] SQL vs R

David R Forrest drf at vims.edu
Tue May 6 16:21:59 CEST 2014


It sounds as if your underlying MySQL database is too slow for your purposes.  Whatever you layer on top of it will be constrained by the underlying database.  To speed up the process significantly, you may need to do work on the database backend part of the process.

Dave


On May 6, 2014, at 7:08 AM, Dr Eberhard Lisse <el at lisse.NA> wrote:

> Thanks,
> 
> tried all of that, too slow.
> 
> el
> 
> on 2014-05-06, 12:00 Gabor Grothendieck said the following:
>> On Tue, May 6, 2014 at 5:12 AM, Dr Eberhard Lisse <el at lisse.na> wrote:
>>> Jeff
>>> 
>>> It's in MySQL, at the moment roughly 1.8 GB, if I pull it into a
>>> dataframe it saves to 180MB. I work from the dataframe.
>>> 
>>> But, it's not only a size issue it's also a speed issue and hence
>>> I don't care what I am going to use, as long as it is fast.
>>> 
>>> sqldf is easy to understand for me but it takes ages.  If
>>> alternatives were roughly similar in speed I would remain with
>>> sqldf.
>>> 
>>> dplyr sounds faster, and promising, but the intrinsic stuff is
>>> way beyond me (elderly Gynaecologist) on the learning curve...
>> 
>> You can create indices in sqldf and that can speed up processing
>> substantially for certain operations.  See examples 4h and 4i on
>> the sqldf home page: http://sqldf.googlecode.com.  Also note that
>> sqldf supports not only the default SQLite backend but also MySQL,
>> h2 and postgresql.  See ?sqldf for info on using sqldf with MySQL
>> and the others.
>> 
> 
> -- 
> Dr. Eberhard W. Lisse  \        / Obstetrician & Gynaecologist (Saar)
> el at lisse.NA            / *     |   Telephone: +264 81 124 6733 (cell)
> PO Box 8421             \     /
> Bachbrecht, Namibia     ;____/
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Dr. David Forrest
drf at vims.edu
804-684-7900w
757-968-5509h
804-413-7125c
#240 Andrews Hall
Virginia Institute of Marine Science
Route 1208, Greate Road
PO Box 1346
Gloucester Point, VA, 23062-1346










-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20140506/e7f5372a/attachment.bin>


More information about the R-help mailing list