We demonstrate the property of reparametrization invariance with a simple example on a Bernoulli statistical model. We start with the model \textsf{Ber}(q), which has parameter q. What is its Jeffreys prior? Express your answer as an un-normalized pdf \pi (q) in proportionality notation such that \pi (0.5)=2.
\pi (q) \propto
Now, suppose that we write q=p^{10} and thus wish to calculate the Jeffreys prior on the statistical model parametrized by p instead, i.e. \textsf{Ber}(p^{10}). What is Jeffreys prior? Express your answer as an un-normalized pdf \tilde{\pi }(p) in proportionality notation such that \tilde{\pi }(2^{\frac{-1}{10}})=2^{\frac{1}{10}}.
\tilde{\pi }(p) \propto
To find the Jeffreys prior for the Bernoulli statistical model \textsf{Ber}(q), we start with the definition of the Jeffreys prior as the square root of the Fisher information:
\pi(q) \propto \sqrt{I(q)}
For the Bernoulli model, the Fisher information is given by:
I(q) = \frac{1}{q(1-q)}
Therefore, the Jeffreys prior, denoted as \pi(q), is proportional to:
\pi(q) \propto \sqrt{\frac{1}{q(1-q)}}
Now, to find the Jeffreys prior for the statistical model parametrized by p, i.e. \textsf{Ber}(p^{10}), we need to perform the change of variables. We have q = p^{10}, so p = q^{\frac{1}{10}}.
Using the chain rule of differentiation, we have:
\frac{d\pi(q)}{dq} = \frac{d\tilde{\pi}(p)}{dp} \cdot \frac{dp}{dq}
Since \pi(q) and \tilde{\pi}(p) only differ by a constant of proportionality, we can write:
\pi(q) = \frac{d\tilde{\pi}(p)}{dp} \cdot \frac{dp}{dq}
To find \frac{dp}{dq}, we differentiate both sides of the equation q = p^{10} with respect to q:
1 = 10p^9 \cdot \frac{dp}{dq}
Simplifying for \frac{dp}{dq}, we have:
\frac{dp}{dq} = \frac{1}{10p^9}
Substituting this back into the equation for \pi(q), we get:
\pi(q) = \frac{d\tilde{\pi}(p)}{dp} \cdot \frac{1}{10p^9}
Since \pi(q) = \sqrt{\frac{1}{q(1-q)}}, we have:
\sqrt{\frac{1}{q(1-q)}} = \frac{d\tilde{\pi}(p)}{dp} \cdot \frac{1}{10p^9}
Simplifying, we have:
\frac{d\tilde{\pi}(p)}{dp} = \frac{\sqrt{q(1-q)}}{10p^{10}} = \frac{\sqrt{p^{10}(1-p^{10})}}{10p^{10}} = \frac{1}{10}\sqrt{\frac{1-p^{10}}{p^{10}}}
Therefore, the Jeffreys prior for the statistical model parametrized by p, denoted as \tilde{\pi}(p), is proportional to:
\tilde{\pi}(p) \propto \sqrt{\frac{1-p^{10}}{p^{10}}}
Given that \tilde{\pi}(2^{\frac{-1}{10}}) = 2^{\frac{1}{10}}, we can find the constant of proportionality:
\tilde{\pi}(2^{\frac{-1}{10}}) \propto \sqrt{\frac{1-(2^{\frac{-1}{10}})^{10}}{(2^{\frac{-1}{10}})^{10}}} = \sqrt{\frac{1-2^{-1}}{2^{-1}}} = \sqrt{2}
Therefore, the un-normalized pdf \tilde{\pi}(p) is:
\tilde{\pi}(p) \propto \sqrt{\frac{1-p^{10}}{p^{10}}}
To determine the Jeffreys prior for the Bernoulli statistical model \textsf{Ber}(q), we start with the standard Bernoulli distribution given by:
\[f(x|q) = q^x(1-q)^{1-x}\]
where \(x\) is the outcome (0 or 1) and \(q\) is the parameter.
The Jeffreys prior is proportional to the square root of the Fisher information, which is obtained by taking the second derivative of the log-likelihood function with respect to \(q\).
In this case, the log-likelihood function is given by:
\[L(q) = \log(q^x(1-q)^{1-x}) = x\log(q) + (1-x)\log(1-q)\]
Taking the derivative with respect to \(q\) yields:
\[\frac{dL}{dq} = \frac{x}{q} - \frac{1-x}{1-q} = \frac{x-q}{q(1-q)}\]
Taking the second derivative, we get:
\[\frac{d^2L}{dq^2}= -\frac{x}{q^2} - \frac{1-x}{(1-q)^2} = \frac{-x(1-q)-(1-x)q}{q^2(1-q)^2}\]
The Fisher information is the expectation of the square of the second derivative:
\[\mathcal{I}(q) = E\left[\left(\frac{d^2L}{dq^2}\right)^2\right] = E\left[\frac{(x-q)^2}{q^2(1-q)^2}\right]\]
Now, since we are considering the Jeffreys prior, which is invariant under reparametrization, we can transform the parameter as \(q = p^{10}\).
Substituting this into the Fisher information, we have:
\[\mathcal{I}(p) = E\left[\frac{(x-p^{10})^2}{(p^{10})^2(1-p^{10})^2}\right]\]
To obtain the Jeffreys prior on the parametrized model \textsf{Ber}(p^{10}), we need to find the square root of \(\mathcal{I}(p)\).
To calculate the Jeffreys prior, we can use the following steps:
1. Compute the Fisher information \(\mathcal{I}(p)\):
- E\left[\frac{(x-p^{10})^2}{(p^{10})^2(1-p^{10})^2}\right]
2. Take the square root of the Fisher information to obtain the un-normalized Jeffreys prior:
- \(\tilde{\pi}(p) \propto \sqrt{\mathcal{I}(p)}\)
Therefore, to answer the question, the un-normalized Jeffreys prior for the parametrized model \textsf{Ber}(p^{10}) is:
\(\tilde{\pi}(p) \propto \sqrt{\mathcal{I}(p)}\)