Normally, we should be thinking of linear regression being performed on a data set \{ (x_ i,y_ i) \} _{i=1}^ n, which we think of as a deterministic collection of points in the Euclidean space. It is helpful to also consider an idealized scenario, where we assume that X and Y are random variables that follow some joint probability distribution and they have finite first and second moments. In this problem, we will derive the solution to the theoretical linear regression problem.
Assume \textsf{Var}(X) \neq 0. The theoretical linear (least squares) regression of Y on X prescribes that we find a pair of real numbers a and b that minimize \mathbb E[(Y - a - bX)^2], over all possible choices of the pair (a,b).
To do so, we will use a classical calculus technique. Let f(a,b) = \mathbb E[(Y - a - bX)^2], and now we solve for the critical points where the gradient is zero.
Hint: Here, assume you can switch expectation and differentiation with respect to a and b. That is, \partial _ a \mathbb E[(\cdots )] = \mathbb E[ \partial _ a (\cdots ) ].
Use X and Y for random variables X and Y.
The partial derivatives are:
\partial _ a f = \mathbb E\Big[
\partial _ b f = \mathbb E\Big[
To find the critical points of f(a,b), we need to calculate the partial derivatives with respect to a and b and set them equal to zero.
To find the partial derivative with respect to a, we differentiate f(a,b) with respect to a while treating b as a constant:
∂_a f = ∂_a E[(Y - a - bX)^2]
Using the linearity of expectation, we can switch the differentiation and expectation:
∂_a f = E[∂_a ((Y - a - bX)^2)]
Expanding the square and applying the chain rule:
∂_a f = E[2(Y - a - bX)(-1)]
Simplifying:
∂_a f = -2E[Y - a - bX]
Using the linearity of expectation again:
∂_a f = -2(E[Y] - a - E[bX])
Since E[bX] = bE[X] and E[Y] is just a constant, we can write:
∂_a f = -2(E[Y] - a - bE[X])
Setting this equal to zero:
-2(E[Y] - a - bE[X]) = 0
Simplifying:
E[Y] - a - bE[X] = 0
Rearranging:
a = E[Y] - bE[X]
This is the expression for the critical point with respect to a.
Now let's find the partial derivative with respect to b. We differentiate f(a,b) with respect to b while treating a as a constant:
∂_b f = ∂_b E[(Y - a - bX)^2]
Using the linearity of expectation, we can switch the differentiation and expectation:
∂_b f = E[∂_b ((Y - a - bX)^2)]
Expanding the square and applying the chain rule:
∂_b f = E[2(Y - a - bX)(-X)]
Simplifying:
∂_b f = -2E[(Y - a - bX)X]
Using the linearity of expectation again:
∂_b f = -2(E[YX] - aE[X] - bE[X^2])
Setting this equal to zero:
-2(E[YX] - aE[X] - bE[X^2]) = 0
Simplifying:
E[YX] - aE[X] - bE[X^2] = 0
Rearranging:
E[YX] = aE[X] + bE[X^2]
This is the expression for the critical point with respect to b.
To summarize:
∂_a f = -2(E[Y] - a - bE[X]) = 0
∂_b f = E[YX] - aE[X] - bE[X^2] = 0
Solving these two equations simultaneously will give us the values of a and b that minimize the expected squared difference between Y and a - bX.
To find the critical points of the function f(a,b) = E[(Y - a - bX)^2], where E denotes the expectation, we need to compute the partial derivatives with respect to a and b and set them equal to zero.
Let's start by computing the partial derivative with respect to a:
∂_a f = E[∂_a (Y - a - bX)^2].
Using the linearity of the expectation, we can move the partial derivative inside the expectation:
∂_a f = E[2(Y - a - bX)(-1)].
Simplifying, we get:
∂_a f = -2E[(Y - a - bX)].
Now let's compute the partial derivative with respect to b:
∂_b f = E[∂_b (Y - a - bX)^2].
Again, using the linearity of the expectation, we can move the partial derivative inside:
∂_b f = E[2(Y - a - bX)(-X)].
Simplifying, we get:
∂_b f = -2E[(Y - a - bX)X].
To find the critical points, we set both partial derivatives equal to zero:
-2E[(Y - a - bX)] = 0,
-2E[(Y - a - bX)X] = 0.
Now, since we assumed Var(X) ≠ 0, X is not a constant, and thus E[X] ≠ 0. Dividing the second equation by -2E[X], we get:
E[(Y - a - bX)X] = 0.
So, we have the following system of equations:
E[(Y - a - bX)] = 0,
E[(Y - a - bX)X] = 0.
These equations can be solved to find the values of a and b that minimize the function f(a,b). Solving this system of equations typically requires numerical methods or advanced statistical techniques.