[R] discrete entropy is not rotation invariant?

[Ingmar Visser]

Hi Context,

It seems to me that the changes in entropy are simply due to the binning.
The usual entropy would be the limit of the discrete entropy when the number
of bins goes to infinity. For the discrete entropy to be a useful
approximation to the entropy the number of bins should be reasonably high.

Hth, Ingmar

> Hello,
> 
> suppose one is forming a probability p(x,y), where the
> x,y
> axes are somewhat accidental and rotation is possible.
>  
> 
> I'm thinking about whether the discrete entropy H(x,y)
> should change
> if the probability is rotated in the x,y plane.
> 
> My current conclusion is that it _does_ change, at
> least if the
> entropy is estimated via bins.    As a simple example,
> suppose 
> the probability mass is concentrated in a single bin
> (that is not
> close to the center of rotation), but it is not a true
> delta function,
> merely concentrated.
> 
> After rotation, the probability mass will often be
> spread into two bins,
> simply because it does not exactly align into one bin.
>  This clearly
> changes the estimated entropy.
> 
> My question:  should one regard this as an artifact of
> the binning,
> or is it correct to consider that the entropy changes?
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html