Consider the linear regression model introduced in the slides and lecture, restated below:
Linear regression model : (\mathbf X_1, Y_1),\ldots , (\mathbf X_ n,Y_ n) \in \mathbb {R}^ d \times \mathbb {R} are i.i.d from the linear regression model Y_ i={\boldsymbol \beta }^\top \mathbf X_ i + \varepsilon _ i, \quad \varepsilon _ i \stackrel{iid}{\sim } \mathcal{N}(0,1) for an unknown {\boldsymbol \beta }\in \mathbb {R}^ d and \mathbf X_ i \sim \mathcal{N}_ d(0,I_ d) independent of \varepsilon _ i.
Suppose that {\boldsymbol \beta }= \mathbf{1} \in \mathbb {R}^ d, which denotes the d-dimensional vector with all entries equal to 1.
What is the mean of Y_1?
\mathbb E[Y_1]=\quad
unanswered
What is the variance of Y_1? (Express your answer in terms of d.)
\textsf{Var}(Y_1)=\quad
The mean of Y_1 can be found by substituting {\boldsymbol \beta }=\mathbf{1} and \mathbf X_1=0 into the linear regression model:
\begin{align*}
\mathbb E[Y_1] &= {\boldsymbol \beta }^\top \mathbf X_1 + \mathbb E[\varepsilon_1] \\
&= \mathbf 1^\top \mathbf 0 + \mathbb E[\varepsilon_1] \\
&= 0 + \mathbb E[\varepsilon_1] \\
&= 0.
\end{align*}
Therefore, the mean of Y_1 is 0.
To find the variance of Y_1, we can use the fact that \varepsilon_i \sim \mathcal{N}(0,1). Since \varepsilon_1 is independent of \mathbf X_1, we have:
\textsf{Var}(Y_1) = \textsf{Var}({\boldsymbol \beta }^\top \mathbf X_1 + \varepsilon_1).
Since \mathbf{1}\in \mathbb{R}^d, we have:
{\boldsymbol \beta }^\top \mathbf X_1 = \mathbf{1}^\top \mathbf{0} = 0.
Therefore, the variance of Y_1 reduces to:
\textsf{Var}(Y_1) = \textsf{Var}(\varepsilon_1) = 1.
Therefore, the variance of Y_1 is 1.
To calculate the mean of Y_1, we use the formula for the expected value of a linear combination of random variables:
\mathbb{E}[Y_1] = \mathbb{E}[\boldsymbol{\beta}^\top \mathbf{X}_1 + \varepsilon_1]
Since \boldsymbol{\beta} = \mathbf{1}, we can rewrite the equation as:
\mathbb{E}[Y_1] = \mathbb{E}[\mathbf{1}^\top \mathbf{X}_1 + \varepsilon_1]
Now, since \mathbf{X}_1 \sim \mathcal{N}_d(0, I_d), we know that the mean of \mathbf{X}_1 is 0. Additionally, since \varepsilon_1 \sim \mathcal{N}(0, 1), the mean of \varepsilon_1 is also 0. Therefore, we can simplify the equation further:
\mathbb{E}[Y_1] = \mathbf{1}^\top \mathbb{E}[\mathbf{X}_1] + \mathbb{E}[\varepsilon_1] = \mathbf{1}^\top \cdot 0 + 0 = 0
So, the mean of Y_1 is 0.
To calculate the variance of Y_1, we again use the formula for the variance of a linear combination of random variables:
\textsf{Var}(Y_1) = \textsf{Var}(\boldsymbol{\beta}^\top \mathbf{X}_1 + \varepsilon_1)
Since \boldsymbol{\beta} = \mathbf{1}, we can rewrite the equation as:
\textsf{Var}(Y_1) = \textsf{Var}(\mathbf{1}^\top \mathbf{X}_1 + \varepsilon_1)
Now, since \mathbf{X}_1 \sim \mathcal{N}_d(0, I_d), the variance of \mathbf{X}_1 is I_d (the d-dimensional identity matrix). Additionally, since \varepsilon_1 \sim \mathcal{N}(0, 1), the variance of \varepsilon_1 is 1. Therefore, we can simplify the equation further:
\textsf{Var}(Y_1) = \mathbf{1}^\top \textsf{Var}(\mathbf{X}_1) \mathbf{1} + \textsf{Var}(\varepsilon_1) = \mathbf{1}^\top I_d \mathbf{1} + 1 = d + 1
So, the variance of Y_1 is d + 1.