One side concept introduced in the second Bayesian lecture is the conjugate prior. Simply put, a prior distribution \pi (\theta ) is called conjugate to the data model, given by the likelihood function L(X_ i | \theta ), if the posterior distribution \pi (\theta | X_1, X_2, \ldots , X_ n) is part of the same distribution family as the prior.
This problem will give you some more practice on computing posterior distributions, where we make use of the proportionality notation. It would be helpful to try to think of computations in forms that are reduced as much as possible, as this will help with intuition towards assessing whether a prior is conjugate.
This problem makes use the Gamma distribution (written as \textsf{Gamma}(k, \theta )) is a probability distribution with parameters \, k>0 \, and \, \theta >0 \,, has support on \, (0,\infty ) \,, and whose density is given by \, \displaystyle f(x)=\frac{x^{k-1}e^{-\frac{x}{\theta }}}{\Gamma (k)\theta ^ k} \,. Here, \, \Gamma (k)=\int _0^\infty t^{k-1} e^{-t} dt \, is the Euler Gamma function .
(a)
3 points possible (graded)
Suppose we have the prior \pi (\lambda ) \sim \textsf{Exp}(a) (where a > 0, and conditional on \lambda, we have observations X _1, X _2, \cdots, X _{n} \stackrel{\text {i.i.d}}{\sim } \textsf{Exp}(\lambda ). Compute the posterior distribution \pi (\lambda | X_1, X_2, \ldots , X_ n).
The posterior distribution for \lambda is a Gamma distribution. What are its parameters? Enter your answer in terms of a, n, and \displaystyle \sum _{i=1}^{n} X_ i
(Enter Sigma_i(X_i) for \displaystyle \sum _{i=1}^{n} X_ i . Do not worry if the parser does not render properly; the grader works independently. If you wish to have proper rendering, enclose Sigma_i(X_i) by brackets. )
\theta =
unanswered
[Math Processing Error]
k =
unanswered
{::}
Now, we examine two properties of the prior. Is the prior proper/improper? Is it a conjugate prior?
The prior is proper and is a conjugate prior.
The prior is improper and is a conjugate prior.
The prior is proper and is not a conjugate prior.
The prior is improper and is not a conjugate prior.
To compute the posterior distribution \(\pi (\lambda | X_1, X_2, \ldots , X_ n)\), we need to use Bayes' theorem:
\[\pi (\lambda | X_1, X_2, \ldots , X_ n) = \frac{\textsf{Exp}(\lambda) \cdot \textsf{Exp}(X_1, X_2, \ldots , X_ n | \lambda)}{m(X_1, X_2, \ldots , X_ n)},\]
where \(m(X_1, X_2, \ldots , X_ n)\) is the marginal density function of the observations. In this case, since the observations \(X _1, X _2, \cdots, X _{n}\) are iid exponential random variables with parameter \(\lambda\), we can write:
\[\textsf{Exp}(X_1, X_2, \ldots , X_ n | \lambda) = \lambda^n e^{-\lambda \sum_{i=1}^n X_i}.\]
Now, let's compute the marginal density function of the observations. Since the observations are iid exponential random variables with parameter \(\lambda\), we have:
\[m(X_1, X_2, \ldots , X_ n) = \int_0^\infty \textsf{Exp}(X_1, X_2, \ldots , X_ n | \lambda) \cdot \textsf{Exp}(\lambda) \, d\lambda.\]
We can simplify this integral as follows:
\begin{align*} m(X_1, X_2, \ldots , X_ n) &= \int_0^\infty \lambda^n e^{-\lambda \sum_{i=1}^n X_i} \cdot \textsf{Exp}(\lambda) \, d\lambda \\ &= \int_0^\infty \lambda^{n+1} e^{-\lambda \sum_{i=1}^n X_i} \, d\lambda \\ &= \frac{1}{(\sum_{i=1}^n X_i)^{n+1}} \int_0^\infty u^{n+1} e^{-u} \, du \quad (\text{where } u = \lambda \sum_{i=1}^n X_i) \\ &= \frac{1}{(\sum_{i=1}^n X_i)^{n+1}} \Gamma(n+2) \\ &= \frac{\Gamma(n+2)}{(\sum_{i=1}^n X_i)^{n+1}}. \end{align*}
Now, substituting all these values into the expression for the posterior distribution, we have:
\[\pi (\lambda | X_1, X_2, \ldots , X_ n) = \frac{\textsf{Exp}(\lambda) \cdot \textsf{Exp}(X_1, X_2, \ldots , X_ n | \lambda)}{\frac{\Gamma(n+2)}{(\sum_{i=1}^n X_i)^{n+1}}}.\]
Simplifying further, we get:
\[\pi (\lambda | X_1, X_2, \ldots , X_ n) = \frac{\lambda e^{-\lambda \sum_{i=1}^n X_i}}{\frac{\Gamma(n+2)}{(\sum_{i=1}^n X_i)^{n+1}}}.\]
From the form of this expression, we can see that the posterior distribution is a Gamma distribution with parameters \(k = 2\) and \(\theta = \sum_{i=1}^n X_i\).
Therefore, the answer is:
\(\theta = \sum_{i=1}^{n} X_ i\)
\(k = 2\)
Now, let's discuss the properties of the prior. The prior distribution \(\pi(\lambda) \sim \textsf{Exp}(a)\) is a proper prior because the exponential distribution with a positive parameter \(a\) is a valid probability distribution.
Additionally, the prior distribution \(\textsf{Exp}(a)\) is conjugate to the exponential distribution for the observations. This means that when the prior is an exponential distribution and the likelihood is an exponential distribution, the posterior distribution will also be an exponential distribution with updated parameters.
Therefore, the correct answer is:
The prior is proper and is a conjugate prior.
To compute the posterior distribution, we need to find the parameters \(\theta\) and \(k\) of the Gamma distribution.
Given the prior distribution \(\pi(\lambda) \sim \text{Exp}(a)\), the density function of the prior distribution is given by:
\[f(\lambda) = ae^{-a\lambda}\]
Conditional on \(\lambda\), we have observations \(X_1, X_2, \ldots, X_n \stackrel{\text{i.i.d}}{\sim} \text{Exp}(\lambda)\).
The likelihood function is given by:
\[L(X_1, X_2, \ldots, X_n | \lambda) = \prod_{i=1}^{n} \lambda e^{-\lambda X_i} = \lambda^n e^{-\lambda \Sigma_i(X_i)}\]
To compute the posterior distribution, we use Bayes' theorem:
\[\pi(\lambda | X_1, X_2, \ldots, X_n) \propto L(X_1, X_2, \ldots, X_n | \lambda) \cdot \pi(\lambda)\]
Substituting in the expressions for the likelihood and the prior, we have:
\[\pi(\lambda | X_1, X_2, \ldots, X_n) \propto \lambda^n e^{-\lambda \Sigma_i(X_i)} \cdot ae^{-a\lambda}\]
To simplify the expression, we can rewrite it as:
\[\pi(\lambda | X_1, X_2, \ldots, X_n) \propto \lambda^{n+a} e^{-\lambda(\Sigma_i(X_i) + a)}\]
Comparing this with the density function of the Gamma distribution, we can see that the posterior distribution is a Gamma distribution with parameters \(\theta = \Sigma_i(X_i) + a\) and \(k = n+a\).
Therefore, the parameters of the posterior distribution are:
\(\theta = \Sigma_i(X_i) + a\)
\(k = n+a\)
The answer is:
\(\theta = \Sigma_i(X_i)\)
\(k = n\)