[R] Calculate Closest 5 Cases?
Albyn Jones
jones at reed.edu
Fri Feb 13 22:32:48 CET 2004
A quick and dirty clustering method (I think its due to Hartigan, at
least I recall first seeing it in his book on clustering) is to pick a
random set of seed cases, and then make one pass through the data,
assigning each case to the seed closest to it. Then you can compute
your distance matrices within the resulting clusters. You could do
this within the resulting clusters again to reduce the size of the
distance matrix computation, and you would need to check neighboring
"clusters" for close points.
albyn
On Fri, Feb 13, 2004 at 01:40:25PM -0500, Tom Blackwell wrote:
> Danny -
>
> The flip answer is, it depends on the size of your computer.
> One can readily calculate the number of entries in the pairwise
> distance matrix that you would like to calculate, and ask whether
> it will fit inside the physical memory installed in your computer.
> It is 50,000 x 50,000 x 8 bytes per floating point number, for
> a total of 20,000,000,000 bytes or 20 gigabytes. The critical
> information that's still missing is that R needs enough space
> for 10 or 20 copies of the largest object in its workspace, in
> order to turn around and assign that object to a new name, or
> do any summaries on it, etc. So, . . . if you have a computer
> with between 200 and 400 gigabytes of random access memory, yes,
> you can calculate and summarize the matrix of pairwise distances.
> But that requires more memory slots than any ordinary motherboard
> provides. (It would be a mother of a motherboard !)
>
> So, failing that, you could always use Adrian Raftery and Chris
> Fraley's 'mclust' package to cluster your data into 50 or more
> clusters of very similar cases (instructions for running mclust()
> on large data sets are found in the manual which comes with the
> package), then calculate all pairwise distances only between the
> cases within each cluster. That's a bit of work to code up.
> You wouldn't want to work interactively for each of 50 clusters.
> But it certainly can be done in R. Depends how much effort you
> want to put into it.
>
> - tom blackwell - u mighigan medical school - ann arbor -
>
> On Fri, 13 Feb 2004 dsheuman at rogers.com wrote:
>
> > I've only begun investigating R as a substitute for SPSS.
> >
> > I have a need to identify for each CASE the closest (or most similar) 5
> > other CASES (not including itself as it is automatically the closest). I
> > have a fairly large matrix (50000 cases by 50 vars). In SPSS, I can use Correlate > Distances to generate a matrix of similarity, but only on a small sample. The entire matrix can not be processed at once due to memory limitations.
> >
> > The data are all percents, so they are easy comparable.
> >
> > Is there any way to do this in R?
> >
> > Below is a small sample of the data (from SPSS) and the desired output.
> >
> > Thanks,
> >
> > Danny
> >
> >
> >
> >
> > *Sample Data.
> > DATA LIST LIST /id(F8) var1(F8.2) var2(F8.2) var3(F8.2) var4(F8.2) var5
> > (F8.2) var6(F8.2) var7(F8.2) var8(F8.2) var9(F8.2) var10(F8.2) var11(F8.2).
> > BEGIN DATA.
> > 10170069 3.51 4.02 6.53 11.05 6.53 8.04 13.57 20.10 11.05 8.55
> > 7.04
> > 10190229 1.89 5.66 4.61 7.62 8.45 13.21 9.50 20.82 16.07 9.36
> > 3.77
> > 10540023 3.40 5.08 3.39 4.52 10.18 14.71 13.56 16.38 9.60 7.89
> > 11.85
> > 10650413 6.64 6.64 3.73 4.70 3.78 13.23 19.82 15.98 12.26 8.48
> > 3.78
> > 10662074 5.11 5.81 4.37 5.11 6.55 14.60 18.97 11.68 10.25 8.75
> > 8.79
> > 10770041 6.43 4.17 6.34 4.26 6.34 4.26 19.11 19.20 14.95 12.77
> > 4.35
> > 11010422 3.14 4.71 6.81 7.85 5.75 6.81 15.18 15.18 13.61 11.00
> > 9.44
> > 11060762 7.03 5.03 6.95 5.99 5.92 12.94 15.01 12.06 11.98 8.06
> > 9.02
> > 11070078 4.61 9.22 4.61 7.94 6.27 12.75 14.02 20.49 7.75 7.75
> > 4.61
> > 11180646 4.48 5.35 6.29 5.42 4.55 11.71 20.74 15.32 14.45 8.09
> > 3.61
> > 11460001 5.71 7.34 6.48 5.68 4.07 10.55 13.83 18.69 12.15 9.76
> > 4.87
> > 11650133 6.00 3.72 6.72 6.00 7.50 17.94 13.44 16.37 13.51 5.15
> > 3.65
> > 11650275 4.02 8.06 6.06 8.10 5.06 8.10 17.16 14.12 12.14 14.12
> > 4.02
> > 11780034 4.25 4.28 5.30 5.33 6.38 14.88 15.96 18.08 14.85 7.48
> > 3.20
> > 11790016 4.40 4.40 5.54 4.40 4.40 10.93 17.67 19.72 13.20 12.13
> > 4.33
> > 12660338 6.60 7.54 5.66 8.49 10.38 11.31 16.06 12.26 8.49 8.49
> > 4.73
> > 12660644 5.51 3.14 3.95 7.09 7.11 14.98 15.72 18.90 9.44 5.50
> > 8.65
> > 12661667 5.44 4.50 5.44 4.50 5.44 12.69 13.63 11.81 9.07 13.68
> > 13.79
> > END DATA.
> >
> > *Output should be:.
> > *.
> > * ID1 CLOSEID1 CLOSEID2 CLOSEID3 CLOSEID4 CLOSEID5.
> > * ID2 CLOSEID1 CLOSEID2 CLOSEID3 CLOSEID4 CLOSEID5.
> > * ID3 CLOSEID1 CLOSEID2 CLOSEID3 CLOSEID4 CLOSEID5.
> > * ID4 CLOSEID1 CLOSEID2 CLOSEID3 CLOSEID4 CLOSEID5.
> > * ID5 CLOSEID1 CLOSEID2 CLOSEID3 CLOSEID4 CLOSEID5.
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
> >
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
--
========================================================================
...armaments were not created chiefly for the protection of the
nations but for their enslavement.
The new political gospel: public office is private graft.
[Mark Twain]
================================================================
http://www.reed.edu/~jones Albyn Jones jones at reed.edu
Reed College, Portland OR 97202 (503)-771-1112 x7418
More information about the R-help
mailing list