Suppose that X_1=\Theta +W_1 and X_2=2\Theta +W_2, where \Theta ,W_1,W_2 are independent standard normal random variables. If the values that we observe happen to be X_1=-1 and X_2=1, then the MAP estimate of \Theta is
To find the Maximum A Posteriori (MAP) estimate of Θ, we need to compute the conditional probability of Θ given the observed values X_1 = -1 and X_2 = 1, denoted as P(Θ | X_1, X_2).
By Bayes' theorem, we have:
P(Θ | X_1, X_2) = (P(X_1, X_2 | Θ) * P(Θ)) / P(X_1, X_2)
Since Θ, W1, and W2 are independent standard normal random variables, we can assume that X1 and X2 are also normally distributed with means Θ and 2Θ, respectively, and unit variances:
P(X_1, X_2 | Θ) = P(X_1 | Θ) * P(X_2 | Θ) = (1/√(2π)) * exp(-0.5 * ((-1 - Θ)^2)) * (1/√(2π)) * exp(-0.5 * ((1 - 2Θ)^2))
Substituting these values into the Bayes' theorem equation, we get:
P(Θ | X_1, X_2) = ((1/√(2π)) * exp(-0.5 * ((-1 - Θ)^2)) * (1/√(2π)) * exp(-0.5 * ((1 - 2Θ)^2)) * P(Θ)) / P(X_1, X_2)
We want to maximize this probability, which is equivalent to maximizing its logarithm:
log(P(Θ | X_1, X_2)) = log(((1/√(2π)) * exp(-0.5 * ((-1 - Θ)^2)) * (1/√(2π)) * exp(-0.5 * ((1 - 2Θ)^2)) * P(Θ)) / P(X_1, X_2))
Simplifying further:
log(P(Θ | X_1, X_2)) = log(P(Θ)) - 0.5 * ((-1 - Θ)^2) - 0.5 * ((1 - 2Θ)^2) - log(P(X_1, X_2))
Since Θ, W1, and W2 are independent standard normal random variables, the prior probability P(Θ) follows a standard normal distribution with mean 0 and variance 1. Therefore, log(P(Θ)) = -0.5 * Θ^2 + constant.
Regarding the term log(P(X_1, X_2)), since we are given the values X1 = -1 and X2 = 1, we can compute this term as well.
Finally, to find the MAP estimate of Θ, we need to find the value of Θ that maximizes the expression log(P(Θ | X_1, X_2)).
It's important to note that this is a general outline of the process, and the specific calculations involved may require further details or numerical methods depending on the values of X1 and X2.
To find the Maximum A Posteriori (MAP) estimate of Θ, we need to calculate the conditional probability distribution of Θ given the observed values of X₁ and X₂, and then find the value of Θ that maximizes this probability distribution.
Let's denote the conditional probability distribution of Θ given X₁ and X₂ as p(Θ | X₁, X₂). According to Bayes' theorem:
p(Θ | X₁, X₂) = p(X₁, X₂ | Θ) * p(Θ) / p(X₁, X₂)
The denominator p(X₁, X₂) is known as the evidence and is a normalizing constant that ensures the total probability sums to 1. It is calculated by integrating the joint probability distribution p(X₁, X₂ | Θ) over all possible values of Θ:
p(X₁, X₂) = ∫ p(X₁, X₂ | Θ) * p(Θ) dΘ
Since Θ, W₁, and W₂ are independent standard normal random variables, we know that the joint probability distribution p(X₁, X₂ | Θ) can be written as a product of the individual normal probability densities:
p(X₁, X₂ | Θ) = p(X₁ | Θ, W₁) * p(X₂ | Θ, W₂)
Substituting the given expressions for X₁ and X₂:
p(X₁, X₂ | Θ) = p(-1 | Θ, W₁) * p(1 | Θ, W₂)
Since W₁ and W₂ are standard normal random variables, their probability densities are given by the standard normal distribution:
p(-1 | Θ, W₁) = (1/√(2π)) * e^(-(θ + w₁ + 1)² / 2)
p(1 | Θ, W₂) = (1/√(2π)) * e^(-(2θ + w₂ - 1)² / 2)
Now, let's assume a prior distribution for Θ, denoted as p(Θ). Since Θ is a standard normal random variable, its prior distribution can also be represented by the standard normal distribution:
p(Θ) = (1/√(2π)) * e^(-θ² / 2)
Substituting these expressions into Bayes' theorem:
p(Θ | X₁, X₂) = [(1/√(2π)) * e^(-(θ + w₁ + 1)² / 2)] * [(1/√(2π)) * e^(-(2θ + w₂ - 1)² / 2)] * [(1/√(2π)) * e^(-θ² / 2)] / p(X₁, X₂)
Now, we need to find the value of Θ that maximizes this probability distribution. Since the numerator is a product of exponentials, it is equivalent to maximizing the exponent of the numerator:
argmax(Θ) [(θ + w₁ + 1)² + (2θ + w₂ - 1)² + θ²]
Expanding and simplifying this expression:
argmax(Θ) [6θ² + 4θ(w₁ + w₂ - 1) + 2(w₁ - w₂) + 2]
To find the maximum, we need to take the derivative of this expression with respect to θ and set it equal to zero:
d/dθ [6θ² + 4θ(w₁ + w₂ - 1) + 2(w₁ - w₂) + 2] = 0
12θ + 4(w₁ + w₂ - 1) = 0
Simplifying:
θ = (1 - w₁ - w₂) / 3
Therefore, the MAP estimate of Θ, given the observed values X₁ = -1 and X₂ = 1, is (1 - w₁ - w₂) / 3.