As in the last video, let X=\Theta +W, where \Theta and W are independent normal random variables and W has mean zero.
a) Assume that W has positive variance. Are X and W independent?
yes or no
b) Find the MAP estimator of \Theta based on X if \Theta \sim N(1,1) and W\sim N(0,1), and evaluate the corresponding estimate if X=2.
\hat\theta =\,
c) Find the MAP estimator of \Theta based on X if \Theta \sim N(0,1) and W\sim N(0,4), and evaluate the corresponding estimate if X=2.
\hat\theta =\,
d) For this part of the problem, suppose instead that X=2\Theta +3W, where \Theta and W are standard normal random variables. Find the MAP estimator of \Theta based on X under this model and evaluate the corresponding estimate if X=2.
\hat\theta =\,
a) No, X and W are not independent since X is a sum of two random variables, \Theta and W. The independence of X and W would require that \Theta and W are uncorrelated and that the distribution of X is the convolution of the distributions of \Theta and W.
b) To find the MAP estimator of \Theta based on X, we need to maximize the posterior probability of \Theta given X. Using Bayes' theorem:
P(\Theta|X) = P(X|\Theta) * P(\Theta) / P(X)
Since \Theta and W are independent and normally distributed, we have:
P(X|\Theta) = P(\Theta + W|\Theta) = P(W) = N(W; 0, 1)
P(\Theta) = N(\Theta; 1, 1^2)
P(X) = \int P(X|\Theta) * P(\Theta) d\Theta
Now, we can use Bayes' theorem:
P(\Theta|X) = (N(W; 0, 1) * N(\Theta; 1, 1^2)) / \int (N(W; 0, 1) * N(\Theta; 1, 1^2)) d\Theta
To maximize the posterior, we need to find the value of \Theta that maximizes P(\Theta|X). Taking the natural logarithm of the numerator, we have:
\log(P(\Theta|X)) = \log(N(W; 0, 1) * N(\Theta; 1, 1^2))
Using properties of logarithms and maximizing with respect to \Theta, we can find the value of \hat\Theta:
\hat\Theta = argmax(\log(N(W; 0, 1) * N(\Theta; 1, 1^2)))
To evaluate the corresponding estimate if X=2, we substitute X=2 into the expression for \hat\Theta and solve for the value.
c) Following similar steps as in part b), the MAP estimator of \Theta based on X, under the given distributions, is:
\hat\Theta = argmax(\log(N(W; 0, 4) * N(\Theta; 0, 1^2)))
To evaluate the corresponding estimate if X=2, we substitute X=2 into the expression for \hat\Theta and solve for the value.
d) With X=2\Theta + 3W, we can follow similar steps as in parts b) and c) to find the MAP estimator of \Theta based on X. The expression for \hat\Theta will be different, but the process is the same. Evaluate the corresponding estimate by substituting X=2 into the expression for \hat\Theta and solving for the value.
a) No, X and W are not independent. Since X is defined as the sum of two random variables, it is dependent on both \Theta and W.
b) The MAP estimator of \Theta based on X is given by:
\hat\theta = argmax P(\Theta|X)
= argmax P(X|\Theta)P(\Theta)
= argmax P(X|\Theta)P(\Theta=1)
Since X = \Theta + W, and W has mean zero, we have X|\Theta = \Theta + W ~ N(\Theta, 1). P(X|\Theta) can be expressed as the likelihood function of X given \Theta.
Likelihood function: L(\theta|X) = P(X|\theta) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-\theta)^2}
P(\Theta=1) is the prior probability distribution of \Theta, which is given as N(1,1).
We can now substitute these values into the MAP estimator:
\hat\theta = argmax \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(\theta-1)^2}
To find the maximum, we take the logarithm of the expression and set the derivative equal to zero:
\frac{d}{d\theta} [\ln(\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(\theta-1)^2})] = 0
Simplifying and solving for \theta, we get:
\hat\theta = \frac{X+1}{2}
Evaluating the estimate for X=2:
\hat\theta = \frac{2+1}{2} = 1.5
c) Following the same steps as in part b, we have:
\hat\theta = argmax \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{8}\theta^2}
To find the maximum, we take the logarithm of the expression and set the derivative equal to zero:
\frac{d}{d\theta} [\ln(\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{8}\theta^2})] = 0
Simplifying and solving for \theta, we get:
\hat\theta = \frac{X}{1+\frac{1}{8}} = \frac{8X}{9}
Evaluating the estimate for X=2:
\hat\theta = \frac{16}{9}
d) Following the same steps as in parts b and c, we have:
X = 2\Theta + 3W
\hat\theta = argmax \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-2\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\theta^2}
To find the maximum, we take the logarithm of the expression and set the derivative equal to zero:
\frac{d}{d\theta} [\ln(\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(X-2\theta)^2} \cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\theta^2})] = 0
Simplifying and solving for \theta, we get:
\hat\theta = \frac{X}{2}
Evaluating the estimate for X=2:
\hat\theta = \frac{2}{2} = 1