[R] Using read.csv.sql() to read in specific columns

[David Reiner]

That seems right to me.

You might want to look into a persistent connection as in Example 10 at
  https://code.google.com/p/sqldf/#Example_10._Persistent_Connections

This will save reloading the DB for every query.

You might even want to create a permanent DB as in Example 9.

This is one of my favorite packages - kudos to Gabor again!
-- David

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Doran, Harold
Sent: Thursday, September 18, 2014 10:38 AM
To: r-help at r-project.org
Subject: [R] Using read.csv.sql() to read in specific columns

I am dealing with data frames that have thousands of columns and hundreds of thousands of rows and only need a few specific columns from the data. The data take various formats, but normally are tab-delimited.

I have written the following which is working as expected. However, because I�m so new at using sqldf(), just looking for some verification from users that this is in fact efficient and correct in the R-ish sense of the word and generalizable to larger data sets.

Harold

tmp <- data.frame(replicate(50, rnorm(10)))
names(tmp) <- paste('item', 1:50, sep='')
write.table(tmp, 'tmp.txt')
read.csv.sql("tmp.txt", sql = "select item1, item2, item50 from file", sep = ' ')

        [[alternative HTML version deleted]]



This e-mail and any materials attached hereto, including, without limitation, all content hereof and thereof (collectively, "XR Content") are confidential and proprietary to XR Trading, LLC ("XR") and/or its affiliates, and are protected by intellectual property laws.  Without the prior written consent of XR, the XR Content may not (i) be disclosed to any third party or (ii) be reproduced or otherwise used by anyone other than current employees of XR or its affiliates, on behalf of XR or its affiliates.

THE XR CONTENT IS PROVIDED AS IS, WITHOUT REPRESENTATIONS OR WARRANTIES OF ANY KIND.  TO THE MAXIMUM EXTENT PERMISSIBLE UNDER APPLICABLE LAW, XR HEREBY DISCLAIMS ANY AND ALL WARRANTIES, EXPRESS AND IMPLIED, RELATING TO THE XR CONTENT, AND NEITHER XR NOR ANY OF ITS AFFILIATES SHALL IN ANY EVENT BE LIABLE FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, CONSEQUENTIAL, SPECIAL AND PUNITIVE DAMAGES, LOSS OF PROFITS AND TRADING LOSSES, RESULTING FROM ANY PERSON’S USE OR RELIANCE UPON, OR INABILITY TO USE, ANY XR CONTENT, EVEN IF XR IS ADVISED OF THE POSSIBILITY OF SUCH DAMAGES OR IF SUCH DAMAGES WERE FORESEEABLE.