[R] Goodness of Fit for Word Frequency Counts

[Thiemo Fetzer]

Dear Mailing list!

sorry to bother you - but maybe you can help me out. I have been searching
and searching for appropriate tests.

I have a huge dataset of loan requests and I have data at portfolio level,
with average portfolio size 200 loans. I want to test whether portfolios are
randomly drawn. The problem is that I have rather qualitative data, namely I
want to characterize whether loans are randomly selected using word counts.

For each loan, I have a "sector", "activity" and "use description". The "use
description" contains about 15 words, the "activity" description is usually
only one or two words.

What I did as of now was to find the word-counts in the overall portfolio,
which is 110,000 loans. From this, I can compute, based on knowing the size
of a team portfolio the expected frequency of certain keywords appearing.
The "sector" variable is categorical and can take only 17 values, whereas in
the overall distribution I found 180 different words as activity
description.

I now wanted to do a type of "goodness of fit" test to see whether the
portfolios are randomly selected or not. I would expect that certain
portfolios are indeed randomly selected, whereas others arent. 

I did a chi^2, Pearson Tukey and G-test of Goodness of Fit. The problem is
that these tests are usually constructed for categorical data - but if I use
the "activity" word-count it need not be categorical. So I am wondering,
whether this is still appropriate? 

I may have a portfolio of 200 loans in which certain words never appear. In
this case, I am not sure which degrees of freedom to look at. Should I use
as prescribed 179 degrees of freedom as I have 180 "categories" - but these
"arent" real categories...

An example may look as follows - the word is on the left, the expected and
observed word counts are given:


     +---------------------------------------+
     |        word      observed   expected  |
     |---------------------------------------|
  1. |           food         54   57.511776 |
  2. |         retail         46    49.04432 |
  3. |    agriculture         39   36.557732 |
  4. |       services         23   15.867387 |
  5. |       clothing         13   14.126975 |
     |---------------------------------------|
  6. | transportation         10   6.5851929 |
  7. |        housing          3     4.65019 |
  8. |   construction          2   4.3173841 |
  9. |           arts          5   4.2500955 |
 10. |  manufacturing          1   3.0170768 |
     |---------------------------------------|
 11. |         health          2    1.751323 |
 12. |            use          0   .55215646 |
 13. |       personal          0   .25241221 |
 14. |      education          2   .68743521 |
 15. |      wholesale          0   .11241227 |
     |---------------------------------------|
 16. |  entertainment          0   .32743521 |
 17. |          green          0   .42743782 |
     +---------------------------------------+

I can have R calculate the Chi² statistic from this, but should I use now 17
degrees of freedom? The problem is this is not categorical data! In this
case, do I have to make comparisons on a word-by-word basis? Like a
"Bernoulli"?

I was looking for other goodness of fit tests for this kind of data for days
now, but I cant really find any others!

I really appreciate your thoughts,

best Thiemo


---
http://freigeist.devmag.net