Hostname: page-component-5db58dd55d-jhf8m Total loading time: 0 Render date: 2026-06-15T18:31:25.651Z Has data issue: false hasContentIssue false

Home
>Journals
>Advances in Applied Probability
>Volume 6 Issue 1
>Why the Shannon and Hartley entropies are ‘natural’

English
Français

Article contents

Why the Shannon and Hartley entropies are ‘natural’

Published online by Cambridge University Press: 01 July 2016

J. Aczél ,

B. Forte and

C. T. Ng

👁 Image
Show author details

J. Aczél: Affiliation:
University of Waterloo, Ontario
B. Forte: Affiliation:
University of Waterloo, Ontario
C. T. Ng: Affiliation:
University of Waterloo, Ontario

Article contents

Get access 👁 Image
Rights & Permissions [Opens in a new window]

Abstract

The following properties of entropies, as measures of expected information, seem natural. The amount of information expected from an experiment does not change if we add outcomes of zero probability (expansibility). The expected information is symmetric in the (probabilities of the) outcomes. The information expected from a combination of two experiments is less than or equal to the sum of the informations expected from the single experiments (subadditivity); equality holds here if the two experiments are independent (additivity).

In this paper it is shown that linear combinations of the Shannon and Hartley entropies and only these have the above properties. The Shannon and the Hartley entropies are also individually characterized.

Keywords

ENTROPIES CHARACTERIZATION THEOREMS PROBABILITY DISTRIBUTIONS INDEPENDENCE RECURSIVITY INFORMATION FUNCTIONS CONTINUOUS CONCAVE INEQUALITIES SUBADDITIVE ADDITIVE NUMBER THEORETICAL FUNCTIONS

Information

Type: Research Article
Information: Advances in Applied Probability , Volume 6 , Issue 1 , March 1974 , pp. 131 - 146

DOI: https://doi.org/10.2307/1426210 [Opens in a new window]
Copyright: Copyright © Applied Probability Trust 1974

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Aczél, J. (1964) Some unsolved problems in the theory of functional equations. Arch. Math. (Basel) 15, 435–444.Google Scholar

Aczél, J. (1968) On different characterizations of entropies. Probability and Information Theory. Proc. Internat. Symp. McMaster Univ., Canada, 1968. Springer-Verlag, Berlin, Heidelberg and New York. pp. 1–11.Google Scholar

Aczél, J. (1970) Problems 6 (P51, P52, P53, P54, P51S1). Aequationes Math. 4, 242–243.Google Scholar

Aczél, J. and Daróczy, Z. (1963) Sur la caractérisation axiomatique des entropies d'ordre positif, y comprise l'entropie de Shannon. C. R. Acad. Sci. Paris 257, 1581–1584.Google Scholar

Aczél, J. and Daróczy, Z. (1974) On Measures of Information and their Characterizations. Academic Press, New York.Google Scholar

Aczél, J. and Forte, B. (1970) A system of axioms for the measure of the uncertainty. Notices Amer. Math. Soc. 17, 202.Google Scholar

Daróczy, Z. (1969) On the Shannon measure of information. (In Hungarian.) Magyar Tud. Akad. III. Oszt. Közl. 19, 9–24. English translation in Selected Translations in Mathematical Statistics and Probability, Vol. 10, 193–210. Inst. of Math. Stat. — Amer. Math. Soc., Providence, R. I., 1972.Google Scholar

Daróczy, Z. (1970) Generalized information functions. Information and Control 16, 36–51.Google Scholar

Faddeev, D. K. (1956) On the concept of entropy of a finite probabilistic scheme. (In Russian.) Uspechi Mat. Nauk 11, No. 1 (67), 227–231.Google Scholar

Forte, B. (1971) Functional inequalities in information theory. Aequationes Math. 6, 102–103.Google Scholar

Forte, B. (1973) Why Shannon's entropy. Convegno Inform. Teor., Ist. Naz. Alta Mat., Roma, 1973; Symposia Math. Vol. XI, Academic Press, New York, 1974.Google Scholar

Hartley, R. V. (1928) Transmission of information. Bell System Tech. J. 7, 535–563.CrossRef Google Scholar

Jaglom, A. M. and Jaglom, I. M. (1957–1969) Probability and Information. (In Russian.) GITTL, Moscow, 1957; 2nd ed., Fizmatgiz, Moscow, 1960. French translation, Probabilité et Information. Collection Sigma Vol. 17, Dunod, Paris, 1969.Google Scholar

Kannappan, Pl. and Ng, C. T. (1973) Measurable solutions of functional equations related to information theory. Proc. Amer. Math. Soc. 38, 303–310.Google Scholar

Kátai, I. (1967) A remark on additive arithmetical functions. Ann. Univ. Sci. Budapest. Eötvös Sect. Math. 12, 81–83.Google Scholar

Kendall, D. G. (1963) Functional equations in information theory. Zeit. Wahrscheinlichkeitsth. 2, 225–229.CrossRef Google Scholar

Lee, P. M. (1964) On the axioms of information theory. Ann. Math. Statist. 35, 414–418.Google Scholar

Rényi, A. (1960) On measures of entropy and information. Proc. 4th Berkeley Symp. Math. Statist. and Prob. 1960. I, 547–561. Univ. of California Press, Berkeley.Google Scholar

Rényi, A. (1965) On the foundations of information theory. Rev. Int. Statist. Inst. 33, 1–14.Google Scholar

Shannon, C. F. and Weawer, W. (1949) The Mathematical Theory of Communication. Univ. of Illinois Press, Chicago.Google Scholar

Tverberg, H. (1958) A new derivation of the information function. Math. Scand. 6, 297–298.Google Scholar

105

URL: https://www.cambridge.org/core/journals/advances-in-applied-probability/article/abs/why-the-shannon-and-hartley-entropies-are-natural/EEAF9012F690E60E9DAB1696672DD37C