Click image to open full size
Claude Shannon: Father of Information Theory
Bits, entropy, and the mathematics of communication
Related Tools
Related Labs
Related Worksheets
Related Cheat Sheets
Claude Shannon was an American mathematician and engineer whose work made modern digital communication possible. In 1948, he published A Mathematical Theory of Communication, which gave scientists a precise way to measure information. His ideas explain how messages can be encoded, transmitted, compressed, and protected from noise. This matters because every phone call, text message, image file, internet packet, and stored bit of data depends on principles Shannon helped create.
Shannon showed that information could be measured in bits and that uncertainty could be described mathematically using entropy. He also connected Boolean algebra to electrical switching circuits, helping lay the foundation for digital computers. His theory identified limits on how much information a communication channel can carry and how errors can be corrected. By linking logic, probability, circuits, and communication, Shannon built a bridge between mathematics and the information age.
Key Facts
- Claude Shannon lived from 1916 to 2001 and is widely called the father of information theory.
- A bit is the information gained from choosing between two equally likely possibilities, such as 0 or 1.
- Shannon entropy measures average uncertainty: H = -sum p_i log2(p_i).
- For N equally likely messages, the information needed is I = log2(N) bits.
- Channel capacity describes the maximum reliable information rate of a noisy channel.
- Shannon showed that Boolean algebra can describe switching circuits, linking logic to digital hardware.
Vocabulary
- Bit
- A bit is the basic unit of digital information and can have one of two values, usually 0 or 1.
- Information theory
- Information theory is the mathematical study of how information is measured, encoded, transmitted, and protected from errors.
- Entropy
- Entropy is a measure of the average uncertainty or information content in a set of possible messages.
- Boolean algebra
- Boolean algebra is a system of logic using values such as true and false or 1 and 0 to describe decisions and circuits.
- Channel capacity
- Channel capacity is the greatest rate at which information can be sent through a communication channel with arbitrarily low error.
Common Mistakes to Avoid
- Thinking a bit is the same as a byte, which is wrong because a byte usually contains 8 bits.
- Treating entropy as disorder only, which is incomplete because Shannon entropy measures uncertainty in possible messages.
- Assuming all messages contain the same amount of information, which is wrong because less probable messages carry more information when they occur.
- Ignoring noise in communication problems, which is wrong because Shannon's theory shows that noise limits reliable transmission and requires coding strategies.
Practice Questions
- 1 A system has 16 equally likely messages. How many bits are needed to identify one message?
- 2 A source sends four symbols with probabilities 1/2, 1/4, 1/8, and 1/8. Calculate the Shannon entropy using H = -sum p_i log2(p_i).
- 3 Explain why Shannon's use of Boolean algebra in switching circuits was important for the development of digital computers.