As in the last video, let X=delta+W, where delta and W are independent normal random variables and W has mean zero.
1. Assume that W has positive variance. Are X and W independent?
2. Find the MAP estimator of delta based on X if delta follows Normal(1,1) and W follows Normal(0,1), and evaluate the corresponding estimate if X=2.
1. No, X and W are not independent. Independence between two random variables means that the joint distribution of the variables can be expressed as a product of their marginal distributions. However, in this case, X = delta + W, and since delta and W are both random variables, X depends on both of them, indicating a dependence between X and W.
2. To find the Maximum A Posteriori (MAP) estimator of delta based on X, we need to compute the posterior distribution of delta given X and then find the value of delta that maximizes this distribution.
First, let's define the prior distribution and likelihood function:
- Prior: delta follows Normal(1, 1)
- Likelihood: X follows Normal(delta, 1)
Using Bayes' rule, the posterior distribution of delta given X can be calculated as:
P(delta|X) ∝ P(X|delta) * P(delta)
Since both the prior and likelihood are normal distributions, we can apply the properties of normal distributions to simplify the calculations. The formula for the posterior distribution is:
P(delta|X) ∝ exp(-0.5 * (X - delta)^2) * exp(-0.5 * (delta - 1)^2)
We can maximize the posterior distribution by finding the value of delta that minimizes the exponent in the equation above. Taking the logarithm of the posterior distribution, we get:
log(P(delta|X)) = -0.5 * (X - delta)^2 - 0.5 * (delta - 1)^2 + constant
Differentiating the logarithm of the posterior distribution with respect to delta and setting it equal to zero, we can find the maximum:
d(log(P(delta|X)))/d(delta) = X - delta + delta - 1 = 0
Simplifying the equation above, we get:
X - 1 = delta
Therefore, the MAP estimator of delta based on X is X - 1.
If X = 2, then the estimate of delta would be:
delta = X - 1 = 2 - 1 = 1
So, the corresponding estimate would be 1.
1. To determine if X and W are independent, we need to check if the joint distribution of X and W can be factorized into the product of their marginal distributions.
Given X = delta + W, we can rewrite this as W = X - delta. Since delta and W are independent, their joint distribution can be expressed as the product of their marginal distributions:
f(x, w) = f(delta)f(w)
Therefore, we need to find the marginal distribution of X and compare it to the conditional distribution of X given W.
The marginal distribution of X can be obtained by integrating out delta:
f(x) = ∫f(delta)f(w) d(delta) = ∫f(delta) d(delta) * f(w) = f(w)
As we can see, the marginal distribution of X is the same as the distribution of W. Therefore, X and W are independent.
2. To find the Maximum A Posteriori (MAP) estimator of delta based on X, we need to maximize the posterior probability of delta given X. The posterior probability can be written as:
P(delta|X) ∝ P(X|delta)P(delta)
Given that delta follows Normal(1,1) and W follows Normal(0,1), the likelihood function P(X|delta) can be expressed as the probability density function (pdf) of a normal distribution with mean delta and variance 1.
P(X|delta) = f(X|delta) = (1/√(2π(1^2))) * exp(-(X-delta)^2/(2*1^2))
The prior probability P(delta) can be expressed as the pdf of a normal distribution with mean 1 and variance 1.
P(delta) = f(delta) = (1/√(2π(1^2))) * exp(-(delta-1)^2/(2*1^2))
To find the MAP estimator of delta, we need to find the value of delta that maximizes the posterior probability P(delta|X). This can be done by taking the logarithm of the posterior probability and finding its maximum.
log(P(delta|X)) = log(P(X|delta)) + log(P(delta))
= -0.5*log(2π) - (X-delta)^2/2 -0.5*log(2π) - (delta-1)^2/2
Taking the derivative of log(P(delta|X)) with respect to delta and setting it equal to zero, we can find the maximum:
d/delta [log(P(delta|X))] = 2*(X-delta) + 2*(delta-1) = 0
2*X - 2*delta + 2*delta - 2 = 0
2*X - 2 = 0
delta = X - 1
Therefore, the MAP estimator of delta based on X is delta = X - 1.
If X = 2, the estimate of delta would be:
delta_estimated = X - 1 = 2 - 1 = 1