In this exercise we want to understand a little better the formula
\frac{1}{\displaystyle {\sum _{i=0}^{n} \frac{1}{\sigma _ i^2}}}
for the mean squared error by considering two alternative scenarios.
In the first scenario, \Theta \sim N(0,1) and we observe X=\Theta +W, where W\sim N(0,1) is independent of \Theta.
In the second scenario, the prior information on \Theta is extremely inaccurate: \Theta \sim N(0,\sigma _0^2), where \sigma _0^2 is so large that it can be treated as infinite. But in this second scenario we obtain two observations of the form X_ i=\Theta +W_ i, where the W_ i are standard normals, independent of each other and of \Theta.
The mean squared error is:
a)smaller in the first scenario.
b)smaller in the second scenario.
c)the same in both scenarios.
To compare the mean squared error in the first and second scenarios, let's calculate the mean squared error for each scenario step-by-step.
First Scenario:
In the first scenario, we have Θ ~ N(0,1) and X = Θ + W, where W ~ N(0,1) is independent of Θ.
The mean squared error (MSE) is given by the formula:
MSE = 1 / (Σ(1 / σᵢ²))
In this case, we have only one observation X, so the formula becomes:
MSE₁ = 1 / (1 / σ₁²)
where σ₁² is the variance of X, which is the sum of the variances of Θ and W:
σ₁² = Var(X) = Var(Θ + W) = Var(Θ) + Var(W) = 1 + 1 = 2
Substituting this into the formula, we get:
MSE₁ = 1 / (1 / 2) = 2
So, in the first scenario, the mean squared error is 2.
Second Scenario:
In the second scenario, we have Θ ~ N(0,σ₀²), where σ₀² is so large that it can be considered infinite. We obtain two observations of the form Xᵢ = Θ + Wᵢ, where the Wᵢ are standard normals, independent of each other and of Θ.
The MSE in this case is given by the same formula:
MSE = 1 / (Σ(1 / σᵢ²))
Since we have two observations X₁ and X₂, the formula becomes:
MSE₂ = 1 / (1 / σ₁² + 1 / σ₂²)
where σ₁² and σ₂² are the variances of X₁ and X₂, respectively. In this scenario, since the observations are from the same distribution, the variances are identical:
σ₁² = σ₂² = Var(Xᵢ) = Var(Θ + Wᵢ) = Var(Θ) + Var(Wᵢ) = σ₀² + 1
Substituting this into the formula, we have:
MSE₂ = 1 / (1 / (σ₀² + 1) + 1 / (σ₀² + 1)) = 1 / (2 / (σ₀² + 1)) = (σ₀² + 1) / 2
As σ₀² approaches infinity, the mean squared error becomes:
MSE₂ = (∞ + 1) / 2 = ∞
So, in the second scenario, the mean squared error is infinite.
Comparing the results, we can see that the mean squared error is smaller in the first scenario (MSE₁ = 2) than in the second scenario (MSE₂ = ∞). Therefore, the correct answer is:
a) Smaller in the first scenario.
To compare the mean squared error in the first and second scenarios, we'll calculate the mean squared error for each scenario and compare them.
In the first scenario, we have X = Θ + W, where Θ ~ N(0,1) and W ~ N(0,1) independent of Θ. The mean squared error is given by the formula:
MSE = E[(Θ - X)^2]
To calculate this, we substitute X = Θ + W into the formula:
MSE = E[(Θ - (Θ + W))^2] = E[(Θ - Θ - W)^2] = E[(-W)^2] = E[W^2] = Var(W) = 1
In the second scenario, we have two observations X1 and X2 of the form X_i = Θ + W_i, where W_i are standard normals and independent of each other and Θ. The mean squared error is again given by the formula:
MSE = E[(Θ - X1)^2] = E[(Θ - X2)^2]
Since Θ ~ N(0, σ0^2) and σ0^2 is treated as infinite, the posterior distribution of Θ given the observations X1 and X2 becomes N(X-bar, σ0^2), where X-bar is the sample mean of X1 and X2.
Using this posterior distribution, the mean squared error can be calculated as:
MSE = E[(Θ - X-bar)^2] = Var(Θ) + E[(X-bar)^2] - 2Cov(Θ, X-bar]
Since Θ ~ N(0, σ0^2), Var(Θ) = σ0^2, and Cov(Θ, X-bar) = Cov(Θ, (X1 + X2)/2) = Cov(Θ, X1)/2 + Cov(Θ, X2)/2 = 0, as Θ and Xi are independent.
Thus, the mean squared error reduces to:
MSE = σ0^2 + E[(X-bar)^2]
Using the fact that Xi = Θ + Wi, we have:
X-bar = (X1 + X2)/2 = (Θ + W1 + Θ + W2)/2 = Θ/2 + (W1 + W2)/2
Substituting this into the mean squared error formula, we get:
MSE = σ0^2 + E[(Θ/2 + (W1 + W2)/2)^2] = σ0^2 + Var(Θ/2 + (W1 + W2)/2) = σ0^2 + Var(Θ/2) + Var((W1 + W2)/2)
Since Θ and Wi are independent, we have:
Var(Θ/2) = (1/4)Var(Θ) = (1/4)σ0^2
Var((W1 + W2)/2) = (1/4)(Var(W1) + Var(W2)) = (1/4)(1 + 1) = 1/2
Therefore, the mean squared error in the second scenario is:
MSE = σ0^2 + (1/4)σ0^2 + 1/2 = (5/4)σ0^2 + 1/2
Comparing the mean squared error in the first and second scenarios, we have:
MSE in first scenario = 1
MSE in second scenario = (5/4)σ0^2 + 1/2
Since σ0^2 is treated as infinite, the mean squared error in the second scenario is much larger than in the first scenario. Therefore, the answer is:
b) smaller in the second scenario.