[R] data frame subset too slow

[Duke]

Hi Jim,

Is this really a problem for me to use [1] instead of [[1]]? Will this 
make it run slower? Also, if I use dat$V1 %in% list$V1, will it be fine?

Anyway, my data and list are basically gene lists (tab delimited):

$ head test.txt
Xkr4    chr1    -    3204562    3661579    3206102    3661429    3    
3204562,3411782,3660632,    3207049,3411982,3661579,
Rp1    chr1    -    4280926    4399322    4283061    4399268    4    
4280926,4341990,4342282,4399250,    4283093,4342162,4342918,4399322,
Rp1_2    chr1    -    4333587    4350395    4334680    4342906    4    
4333587,4341990,4342282,4350280,    4340172,4342162,4342918,4350395,
Sox17    chr1    -    4481008    4486494    4481796    4483487    5    
4481008,4483180,4483852,4485216,4486371,    
4482749,4483547,4483944,4486023,4486494,
Mrpl15    chr1    -    4763278    4775807    4764532    4775758    5    
4763278,4767605,4772648,4774031,4775653,    
4764597,4767729,4772814,4774186,4775807,
Mrpl15_2    chr1    -    4763278    4775807    4775807    4775807    
4    4763278,4767605,4772648,4775653,    4764597,4767729,4772814,4775807,
$ head list.txt
GeneNames    Chr    Start    End
0610007C21Rik    chr5    31351012    31356996
0610007L01Rik    chr5    130695613    130719635
0610007L01Rik_2    chr5    130698204    130719635
0610007P08Rik    chr13    63916627    64001609
0610007P08Rik_2    chr13    63916641    63970963
0610007P14Rik    chr12    87156404    87165495

Thanks,

D.

On 12/30/10 11:13 AM, jim holtman wrote:
> You should be using dat[[1]].  Here is an example with 80000 rows that
> take about 0.02 seconds to get the subset.
>
> Provide an 'str' of what your data looks like
>
>> n<- 80000  # rows to create
>> dat<- data.frame(sample(1:200, n, TRUE), runif(n), runif(n), runif(n), runif(n))
>> lst<- data.frame(sample(1:100, n, TRUE), runif(n), runif(n), runif(n), runif(n))
>> str(dat)
> 'data.frame':   80000 obs. of  5 variables:
>   $ sample.1.200..n..TRUE.: int  39 116 69 163 51 125 144 32 28 4 ...
>   $ runif.n.              : num  0.519 0.793 0.549 0.77 0.272 ...
>   $ runif.n..1            : num  0.691 0.89 0.783 0.467 0.357 ...
>   $ runif.n..2            : num  0.705 0.254 0.584 0.998 0.279 ...
>   $ runif.n..3            : num  0.873 1 0.678 0.702 0.455 ...
>> str(lst)
> 'data.frame':   80000 obs. of  5 variables:
>   $ sample.1.100..n..TRUE.: int  38 83 38 70 77 44 81 55 32 1 ...
>   $ runif.n.              : num  0.0621 0.7374 0.074 0.4281 0.0516 ...
>   $ runif.n..1            : num  0.879 0.294 0.146 0.884 0.58 ...
>   $ runif.n..2            : num  0.648 0.745 0.825 0.507 0.799 ...
>   $ runif.n..3            : num  0.2523 0.1679 0.9728 0.0478 0.0967 ...
>> system.time({
> + dat.sub<- dat[dat[[1]] %in% lst[[1]],]
> + })
>     user  system elapsed
>     0.02    0.00    0.01
>> str(dat.sub)
> 'data.frame':   39803 obs. of  5 variables:
>   $ sample.1.200..n..TRUE.: int  39 69 51 32 28 4 69 3 48 69 ...
>   $ runif.n.              : num  0.5188 0.5494 0.2718 0.5566 0.0893 ...
>   $ runif.n..1            : num  0.691 0.783 0.357 0.619 0.717 ...
>   $ runif.n..2            : num  0.705 0.584 0.279 0.789 0.192 ...
>   $ runif.n..3            : num  0.873 0.678 0.455 0.843 0.383 ...
> On Thu, Dec 30, 2010 at 10:23 AM, Duke<duke.lists at gmx.com>  wrote:
>> Hi all,
>>
>> First I dont have much experience with R so be gentle. OK, I am dealing with
>> a dataset (~ tens of thousand lines, each line ~ 10 columns of data). I have
>> to create some subset of this data based on some certain conditions (for
>> example, same first column with another dataset etc...). Here is how I did
>> it:
>>
>> # import data
>> dat<- read.table( "test.txt", header=TRUE, fill=TRUE, sep="\t" )
>> list<- read.table( "list.txt", header=TRUE, fill=TRUE, sep="\t" )
>> # create sub data
>> subdat<- dat[dat[1] %in% list[1],]
>>
>> So the third line is to create a new data frame with all the same first
>> column in both dat and list. There is no problem with the code as it runs
>> just fine with testing data (small). When I tried with my real data (~80k
>> lines, ~ 15MB size), it takes like forever (few hours). I dont know why it
>> takes that long, but I think it shouldnt. I think even with a for loop in
>> C++, I can get this done in say few minutes.
>>
>> So anyone has any idea/advice/suggestion?
>>
>> Thanks so much in advance and Happy New Year to all of you.
>>
>> D.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>