004. Entropies and uncertainty of uncertainty

Entropy was born in the 19th century as a daughter of energy. Clausius, Boltzmann and Gibbs (and others) developed the physical notion of entropy. In the 20th century, Hartley (1928) and Shannon (1948) introduced a logarithmic measure of information in electronic communication in order “to eliminate the psychological factors involved and to establish a measure of information in terms of purely physical quantities”. Information theory focused on entropy as a measure of uncertainty of subjective choice. This understanding of entropy was returned from information theory to statistical mechanics by Jaynes as a basis of “subjective” statistical mechanics. The entropy maximum approach was declared as a minimization of the subjective uncertainty. This approach gave rise to a MaxEnt “anarchism”. Many new entropies were invented and now there exists rich choice of entropies for fitting needs. The most celebrated of them are the Renyi entropy, the Burg entropy, the Tsallis entropy and the Cressie–Read family. All of them belong to the class of Csiszár-Morimoto divergences [10]. MaxEnt approach is conditional maximization of entropy for the evaluation of the probability distribution when our information is partial and incomplete. The entropy function may be the classical BGS entropy or any function from the rich family of non-classical entropies. This rich choice causes a new problem: which entropy is better for a given class of applications? The MaxEnt “anarchism” was criticized many times as a “senseless fitting”, nevertheless, it remains a very popular approach to multidimensional problems with uncertainty. We understand entropy as a measure of uncertainty which increases in Markov processes [2]. In this talk, we review the families of non-classical entropies and discuss the question: is entropy a function or an order? We describe the most general ordering of the distribution space, with respect to which all continuous-time Markov processes are monotonic (the Markov order [2]). For inference, this approach results in a convex compact set of conditionally “most random” distributions. The «uncertainty of uncertainty» (a set of distributions instead of a specific distribution) is unavoidable in analysis of non-equilibrium systems.
References

1. I. Csiszár. (1978) Information measures: a critical survey. In: Transactions of the Seventh Prague Conference on Information Theory, Statistical Decision Functions and the Eighth European Meeting of Statisticians, Prague, Czech Republic, 18 August–23 August 1974; Academia: Prague, Czech Republic, 1978; Volume B, pp. 73–86.

2.  A.N. Gorban, P.A. Gorban, G. Judge. (2010) Entropy: The Markov Ordering Approach. Entropy. 2010; 12(5):1145-1193.