Discrete source S₁ has 4 equiprobable symbols while discrete source S₂ has 16 equiprobable symbols. When the entropy of these two sources is compared, the entropy of:

Concept:

Entropy of mutual information H(X):

\(H\left( X \right) = \mathop \sum \nolimits_{i = 1}^n p\left( {{x_i}} \right).{\log _2}\frac{1}{{p\left( {{x_i}} \right)}} = {\log _2}m\)

[for equiprobable ‘m’ no. of symbol]

Calculation:

Case-I: For source S₁;

No. of symbol; m = 4

H(x) = log₂ m

= 2 bits / symbol

Case-II: For source S₂;

No. of symbol; m = 16

H(x) = log₂ m

= 4 bits / symbol

∴ entropy of S₁ is less than S₂

Important Points

1) Entropy for a continuous random variable is known as differential entropy:

\(H\left( X \right) = \mathop \smallint \nolimits_{ - \infty }^\infty {F_X}\left( x \right).{\log _2}\frac{1}{{{F_X}\left( x \right)}}\) 

F_X(x) ⇒ PDF of RV ‘X’

2) Entropy is a measure of uncertainty.