Information Theory and its Applications
Aiden A. Bruen, Mario Professor Forcinito, James Professor McQuillan · 2021
This chapter discusses the main results from information theory relating to source coding and channel capacity for transmission. It explains a connection between weighing problems and information theory. Information theory has several applications. The chapter shows how information theory can be used in a fundamental way in cryptography. The word “entropy” was first coined by Clausius in physics around 1865. Entropy is used mainly in statistical mechanics and thermodynamics. Roughly, it measures the amount of energy lost to heat in an irreversible physical process. Shannon’s first theorem shows that the entropy of a given source, measured in Shannon bits, is arbitrarily close to the number of physical (0, 1) bits on average needed for efficient encoding of the source. Shannon’s second theorem shows that the maximum value of a conditional entropy, i.e. the channel capacity measured in Shannon bits is the maximum transmission rate, measured in physical bits, for accurate communication across the channel.