[BioC] RdbiPgSQL: slowness in dbWriteTable
jgentry at jimmy.harvard.edu
Tue Sep 4 20:56:28 CEST 2007
Hi there ...
In the past I've had some difficulty inserting some larger tables into
postgres using dbWriteTable() from RdbiPgSQL, in terms of speed
performance but now I've run into some data.frames which are just
too large (although they don't seem to be all that large in practical
As an example, I was attempting to insert the assayData from two
SnpCallSets into a DB over the weekend, 6 tables each of ~350 columns and
250K rows. When I came back this morning (~3.5 days), it hadn't even
completed one of these tables, with both CPU and RAM maxxed (on a machine
w/ 16GB of RAM and a pretty decent CPU).
Anyone have ideas on a faster method to get these tables inserted? One
possibility that I've found in the past to be quicker is to manually loop
INSERT statements, but the problem there is detection of data types (which
I believe is the real slowdown from the RdbiPgSQL package in the first
place) and proper quoting/non-quoting.
More information about the Bioconductor