[R] deviance vs entropy
remoteapl at obninsk.com
Fri Feb 16 01:24:46 CET 2001
Thank you for your answer. It gave some food to my brain. Let me ask more...
> I'm not quite sure what you have in mind, but I'm inferring from your
comments that by "deviance"
> you mean:
> -SUM p_i log (p_i/q_i) (or -2 SUM p_i log (p_i/q_i))
I am sorry for my language. I meant in particular those deviance which is
we select split of a node building classification tree. As far as I know it
-2 SUM n_i log (p_i)
where n_i is number of points of class c_i at this node and p_i=n_i/N, where
N is total number
of cases at this node. Probably it refers some way to what you wrote above.
May you tell more
on p_i and q_i in your formula?
> D(p_i||q_i) = - SUM p_i log p_i + SUM p_i log q_i = H(p) - H(p:q)
> where H(p) is entropy of p, and H(p:q) is the cross entropy. If q is the
uniform distribution, then
> the cross entropy reduces to:
I probably understand this and the next statements if I understand the first
> I'm guessing that in the things you've read, when they are talking about
deviance, q can (and
> generally is) something other than the uniform distribution. For example,
p is often the empirical
> distribution of a data sample, and q is the distribution corresponding to
some induced model. Then
> D(p||q) is a measure of how far the model is from the observed data.
It sounds interesting. May you please repeat this in terms of classification
trees? I mean what is
"induced model" and "corresponding distribution" if we are speaking on CART?
> entropy (entropy - cross_entropy, or KL-divergence). Statisticians are
interested in deviance
What "KL" stands for?
> because (with the factor of 2) it is asymptotically chi-square for many
modeling families. In
That's probably the most important argument PRO.
> information theoretic terms it's nice to think of the deviance as the
number of bits extra that it
> would take to transmit the data for a system assuming the distribution q,
relative to a system that
> had assumed p, which is the best system for transmitting that particular
Very interesting! I must think over this more.
> Then again, maybe I've misunderstood you completely. Please set me
straight if I have.
I see that you sit quite straight. I am afraid that I lie horizontally:-)
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
More information about the R-help