Suppose that X_1=delta+W_1 and X_2=2*delta+W_2, where delta, W_1, W_2 are independent standard normal random variables. If the values that we observe happen to be X_1=-1 and X_2=1, then the MAP estimate of delta is?
To find the Maximum A Posteriori (MAP) estimate of delta, we need to use Bayes' theorem and compute the posterior distribution of delta given the observed values of X_1 and X_2.
First, let's find the joint distribution of X_1 and X_2:
X_1 = delta + W_1 (1)
X_2 = 2*delta + W_2 (2)
Since delta, W_1, and W_2 are independent standard normal random variables, we know that W_1 and W_2 are also standard normal random variables.
Now, let's write the joint distribution of X_1 and X_2 using the Gaussian distribution:
P(X_1, X_2 | delta) = P(delta + W_1 = X_1) * P(2*delta + W_2 = X_2)
= (1/sqrt(2*pi)) * exp(-(X_1 - delta)^2 / 2) * (1/sqrt(2*pi)) * exp(-(X_2 - 2*delta)^2 / 2)
Next, we need to determine the prior distribution of delta. Without any additional information, we assume a standard normal prior for delta as well.
P(delta) = (1/sqrt(2*pi)) * exp(-delta^2 / 2)
Now, we can use Bayes' theorem to find the posterior distribution of delta given X_1 and X_2:
P(delta | X_1, X_2) ∝ P(X_1, X_2 | delta) * P(delta)
With the joint distribution and prior distribution expressed above, we have:
P(delta | X_1, X_2) ∝ (1/sqrt(2*pi)) * exp(-(X_1 - delta)^2 / 2) * (1/sqrt(2*pi)) * exp(-(X_2 - 2*delta)^2 / 2) * (1/sqrt(2*pi)) * exp(-delta^2 / 2)
Now, we can simplify the expression by taking the natural logarithm (logarithm is a monotonically increasing function):
log(P(delta | X_1, X_2)) ∝ -(X_1 - delta)^2 / 2 - (X_2 - 2*delta)^2 / 2 - delta^2 / 2
To find the MAP estimate, we need to find the value of delta that maximizes this expression. We can do that by taking the derivative with respect to delta and setting it equal to 0:
d(log(P(delta | X_1, X_2))) / d(delta) = (X_1 - delta) + 2*(X_2 - 2*delta) - delta = 0
Simplifying,
X_1 - delta + 2*X_2 - 4*delta - delta = 0
-6*delta + X_1 + 2*X_2 = 0
Solving for delta, we get:
delta = (X_1 + 2*X_2) / 6
Therefore, the MAP estimate of delta is (X_1 + 2*X_2) / 6.
In this case, X_1 = -1 and X_2 = 1. Plugging these values into the equation, we get:
MAP estimate of delta = (-1 + 2*1) / 6 = 1/6
To find the MAP (Maximum A Posteriori) estimate of delta, we need to consider the conditional probability distribution of delta given the observed values X1 and X2. Let's assume that delta follows a standard normal distribution.
The probability density function (PDF) of X1 can be written as:
f(X1) = (1/sqrt(2*pi))*exp(-0.5*X1^2)
Similarly, the PDF of X2 can be written as:
f(X2) = (1/sqrt(2*pi))*exp(-0.5*X2^2)
Now, we can write the joint probability density function (joint PDF) of X1 and X2 as the product of their respective PDFs, since they are assumed to be independent:
f(X1, X2) = f(X1)*f(X2)
Substituting the values X1 = -1 and X2 = 1, we get:
f(-1, 1) = f(-1)*f(1)
Now, let's express X1 and X2 in terms of delta and W1, W2:
X1 = delta + W1
X2 = 2*delta + W2
Substituting these expressions, we get:
f(-1, 1) = f(delta + W1)*f(2*delta + W2)
Since delta, W1, and W2 are independent standard normal random variables, their PDFs are given by the standard normal distribution:
f(delta) = (1/sqrt(2*pi))*exp(-0.5*delta^2)
f(W1) = (1/sqrt(2*pi))*exp(-0.5*W1^2)
f(W2) = (1/sqrt(2*pi))*exp(-0.5*W2^2)
Now, we can write the joint PDF of delta, W1, and W2 as the product of their respective PDFs:
f(delta, W1, W2) = f(delta)*f(W1)*f(W2)
Substituting these expressions, we get:
f(-1, 1) = (1/sqrt(2*pi))*exp(-0.5*delta^2)*(1/sqrt(2*pi))*exp(-0.5*W1^2)*(1/sqrt(2*pi))*exp(-0.5*W2^2)
Simplifying, we get:
f(-1, 1) = (1/(2*pi))*exp(-0.5*delta^2)*exp(-0.5*W1^2)*exp(-0.5*W2^2)
Now, to find the MAP estimate of delta, we need to find the value of delta that maximizes this joint PDF. Since the term (1/(2*pi)) is a constant, we can ignore it for the maximization:
f(-1, 1) ∝ exp(-0.5*delta^2)*exp(-0.5*W1^2)*exp(-0.5*W2^2)
Taking the logarithm of both sides:
ln(f(-1, 1)) ∝ -0.5*delta^2 - 0.5*W1^2 - 0.5*W2^2
Since we are interested in maximizing this expression, we can ignore any terms that do not depend on delta:
ln(f(-1, 1)) ∝ -0.5*delta^2
Therefore, to find the MAP estimate of delta, we need to find the value of delta that minimizes the expression -0.5*delta^2. This corresponds to maximizing the exponent term, which happens when delta is 0.
Therefore, the MAP estimate of delta, given the observed values X1 = -1 and X2 = 1, is delta = 0.