The mutual information I(X,Y)=H(X)−H(X|Y) between two random variables X and Y satisfies
I(X,Y)>0
I(X,Y)≥0
I(X,Y)≥0, equality holds when X and Y are uncorrelated
I(X,Y)≥0 , equality holds when X and Y are independent
The correct answer is: I(X,Y)≥0, equality holds when X and Y are independent.
To understand why, let's break down the terms in the mutual information formula:
- H(X) is the entropy of random variable X, which is a measure of the uncertainty or information content of X. It tells us how much information is gained on average when observing X. Since entropy can never be negative, H(X)≥0.
- H(X|Y) is the conditional entropy of X given Y, which measures the remaining uncertainty of X when we already know Y. It tells us how much information about X is left once we have observed Y. Like entropy, conditional entropy is also non-negative, so H(X|Y)≥0.
- I(X,Y) is the mutual information between X and Y, which quantifies the amount of information that X and Y share. It calculates the reduction in uncertainty of X when we have information about Y. Mutual information can be both positive and negative, but it can never be less than zero. This means I(X,Y)≥0.
The equality holds when X and Y are independent. In this case, knowing Y provides no information about X, and therefore the conditional entropy H(X|Y) becomes equal to the entropy H(X). This results in I(X,Y)=H(X)−H(X|Y)=H(X)−H(X)=0.
However, it's worth noting that if there is a strong correlation between X and Y (but not independence), the mutual information can still be positive, indicating that there is some shared information.