What is information? Data, discreteness and the bit
Data are signs we can write down: letters, numbers, pixels, sounds. They become information when they tell us something new and reduce our doubt.
A bit is the smallest piece of information: the answer to one fair yes/no question, written 0 or 1. A byte is 8 bits.
Discrete and continuous
A continuous signal, like a sound wave, can take any value. A discrete signal takes only separate values, like steps. To store a wave on a computer we measure it at times (sampling) and round each value to one of a few levels (quantising). With 2 levels each sample needs 1 bit; with 16 levels, 4 bits.
Information processes: storage, processing, transfer
Information goes through four kinds of steps.
- Creating / collecting: a sensor, a camera or a person produces data.
- Storage: keeping data for later (memory, disk, paper).
- Processing: changing data by rules: sorting, calculating, compressing, searching.
- Transfer (transmission): sending data from a source through a channel (cable, radio, fibre) to a receiver.
Real channels have noise that can flip bits. Systems add check bits or repeat the message so the receiver can spot and fix errors. A channel also has a limit on speed, called its bandwidth or bit rate (bits per second).
Measuring information: Hartley's formula
Suppose there are N equally likely outcomes. Each yes/no question can halve the choices. So the number of questions needed is the power of 2 that gives N:
I = log₂ N bits (Hartley's formula)
8 choices: log₂ 8 = 3 bits. 64 choices: 6 bits. If N is not a power of 2 the answer is a fraction, like log₂ 6 ≈ 2.58 bits for a die, and you round up to whole yes/no questions.
For a message of k symbols from an alphabet of N symbols: total information = k × log₂ N bits.
Shannon's entropy: unequal probabilities
If outcomes are not equally likely, a rare outcome surprises us more. The information of one outcome with probability p is −log₂ p bits (p = 1/2 gives 1 bit, p = 1/8 gives 3 bits). The average information per symbol, called entropy, is:
H = −Σ p·log₂ p (Shannon's formula)
For probabilities 1/2, 1/4, 1/8, 1/8: H = 0.5·1 + 0.25·2 + 0.125·3 + 0.125·3 = 1.75 bits. For four equal outcomes H = 2 bits, which equals Hartley's value. Equal probabilities give the largest H. A sure outcome (p = 1) gives H = 0.
Why it matters: if H is small, the message can be compressed to about H bits per symbol.
Try it: twenty questions
Ask a friend to think of a number from 1 to 64. Find it with yes/no questions only. Can you do it in 6? (Hint: always ask 'Is it in the top half?'.) Now try with the numbers 1 to 100. How many questions do you need at most?
Key formulas and definitions
- 1 byte = 8 bits
- Hartley: I = log₂ N bits (N equally likely outcomes)
- Message of k symbols from an N-symbol alphabet: I = k · log₂ N
- Information of one outcome: i = −log₂ p
- Shannon: H = −Σ pᵢ log₂ pᵢ bits per symbol
- Transfer time = amount of data ÷ bit rate
Worked examples
1. A ball is in one of 64 boxes. How many bits of information find it?
I = log₂ 64 = 6 bits (6 yes/no questions).
2. A message has 20 letters from a 32-letter alphabet. How many bits is it?
Each letter gives log₂ 32 = 5 bits. Total = 20 × 5 = 100 bits.
3. How many bits are in an 8-level grey pixel, and in a 10 × 10 image of such pixels?
One pixel: log₂ 8 = 3 bits. Image: 100 × 3 = 300 bits.
4. Find the entropy of a source with probabilities 1/2, 1/4, 1/8, 1/8.
H = ½·1 + ¼·2 + ⅛·3 + ⅛·3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits.
5. Find the information in an event of probability 1/16.
i = −log₂(1/16) = log₂ 16 = 4 bits.
6. How long does it take to send a 10 MB file over a 2 Mbit/s channel? (1 MB = 8 Mbit)
10 MB = 80 Mbit. Time = 80 ÷ 2 = 40 s.
7. How much information does a fair die roll give?
log₂ 6 ≈ 2.58 bits.
Common mistakes
- Mixing up bit and byte. A byte is 8 bits.
- Using Hartley's formula when outcomes are not equally likely. Use Shannon's.
- Thinking more data always means more information. Repeated or predictable data adds little.
- Forgetting the minus sign in −Σ p log₂ p. Since log₂ p is negative for p < 1, the minus makes H positive.