Consider a discrete memory less source with alphabet S = s₀, s₁, s₂, s₃, s₄, ...  and respective probabilities of occurrence P = {1/2, 1/4, 1/8, 1/16, 1/32, .......}. The entropy of the source (in bits) is _______.

Concept: The entropy of the source is the average information. 

Calculation: The entropy of the source is:

 $\mathrm{H}(\mathrm{s})=-{\mathrm{s}}_0\mathrm{l}\mathrm{o}{\mathrm{g}}_2{\mathrm{s}}_0-{\mathrm{s}}_1\mathrm{l}\mathrm{o}{\mathrm{g}}_2{\mathrm{s}}_1-{\mathrm{s}}_2\mathrm{l}\mathrm{o}{\mathrm{g}}_2{\mathrm{s}}_2\dots \dots \dots \dots$

$\Rightarrow \mathrm{H}\left(\mathrm{s}\right)=\frac{1}{2}\times 1+\frac{1}{4}\times 2+\frac{1}{8}\times 3+\frac{1}{16}\times 4.\dots \dots \dots ..$

$\Rightarrow \mathrm{H}\left(\mathrm{s}\right)=\frac{1}{2}\left[1+2\times \frac{1}{2}+3\times \frac{1}{2^2}+4\times \frac{1}{2^3}\dots \dots \dots \dots .\right]$

Now, the expansion of

${\left(1-\mathrm{x}\right)}^{-2}=1+2\mathrm{x}+3{\mathrm{x}}^2+4{\mathrm{x}}^3\dots \dots \dots .$

Comparing with

$1+2\times \frac{1}{2}+3\times \frac{1}{2^2}+4\times \frac{1}{2^3}$

We see, $\mathrm{x}=\frac{1}{2}$

Thus, 

$\begin{array}{l}={\left(1-\frac{1}{2}\right)}^{-2}\\ ={\left(\frac{1}{2}\right)}^{-2}=2^2=4\end{array}$

Now,

H(s) = 0.5 × 4 = 2 bits/symbol