Fisher信息与最大似然估计的渐进正态性（附有在Bernoulli分布上的计算）

写在前面

最大似然估计具有很多好的性质，包括相合性，同变性，渐进正态性等。本文主要关注的是渐进正态性。渐近正态性表明，估计量的极限分布是正态分布。而该正态分布的方差，与Fisher信息有着密不可分的关系。

Fisher信息

（定义）记分函数（Score Function):
$$s(X;\theta )=\frac{\partial logf(X;\theta )}{\partial \theta }.$$
（定义）Fisher信息量(Fisher Information):
I n ( θ ) = V ( ∑ i = 1 n s ( X i ; θ ) ) = ∑ i = 1 n V ( s ( X i ; θ ) )

In​(θ)​=V(i=1∑n​s(Xi​;θ))=i=1∑n​V(s(Xi​;θ))​
（定理）
$${\mathbb{E}}_{\theta }[s(X;\theta )]=0$$
证明：
E θ [ s ( X ; θ ) ] = ∫ x ∂ l o g f ( x ; θ ) ∂ θ f ( x ; θ ) d x = ∫ x 1 f ( x ; θ ) ∂ f ( x ; θ ) ∂ θ f ( x ; θ ) d x = ∫ x ∂ f ( x ; θ ) ∂ θ d x = ∂ ∂ θ ∫ x f ( x ; θ ) d x = ∂ ∂ θ 1 = 0

Eθ[s(X;θ)]=∫x∂logf(x;θ)∂θf(x;θ)dx=∫x1f(x;θ)∂f(x;θ)∂θf(x;θ)dx=∫x∂f(x;θ)∂θdx=∂∂θ∫xf(x;θ)dx=∂∂θ1=0" role="presentation" style="position: relative;">Eθ[s(X;θ)]=∫x∂logf(x;θ)∂θf(x;θ)dx=∫x1f(x;θ)∂f(x;θ)∂θf(x;θ)dx=∫x∂f(x;θ)∂θdx=∂∂θ∫xf(x;θ)dx=∂∂θ1=0$$\begin{array}{rl}{\mathbb{E}}_{\theta }[s(X;\theta )]& ={\int }_x\frac{\mathrm{\partial }logf(x;\theta )}{\mathrm{\partial }\theta }f(x;\theta )dx\\ & ={\int }_x\frac{1}{f(x;\theta )}\frac{\mathrm{\partial }f(x;\theta )}{\mathrm{\partial }\theta }f(x;\theta )dx\\ & ={\int }_x\frac{\mathrm{\partial }f(x;\theta )}{\mathrm{\partial }\theta }dx\\ & =\frac{\mathrm{\partial }}{\mathrm{\partial }\theta }{\int }_{x}f(x;\theta )dx=\frac{\mathrm{\partial }}{\mathrm{\partial }\theta }1\\ & =0\end{array}$$

Eθ​[s(X;θ)]​=∫x​∂θ∂logf(x;θ)​f(x;θ)dx=∫x​f(x;θ)1​∂θ∂f(x;θ)​f(x;θ)dx=∫x​∂θ∂f(x;θ)​dx=∂θ∂​∫x​f(x;θ)dx=∂θ∂​1=0​
（定理）若$f(X;\theta )$二阶可导，则Fisher信息矩阵可以写为如下形式：
$$I_n(\theta )=nI(\theta )=-n{\int }_x\frac{{\partial }^{2}logf(x;\theta )}{\partial {\theta }^2}f(x;\theta )dx$$
证明：
V θ [ s ( X ; θ ) ] = E θ [ s ( X ; θ ) 2 ] − E θ [ s ( X ; θ ) ] 2 = E θ [ s ( X ; θ ) 2 ] = ∫ x ∂ l o g f ( x ; θ ) ∂ θ ∂ l o g f ( x ; θ ) ∂ θ f ( x ; θ ) d x ∫ x ∂ 2 l o g f ( x ; θ ) ∂ θ 2 f ( x ; θ ) d x = ∫ x ∂ ∂ θ ( 1 f ( x ; θ ) ∂ f ( x ; θ ) ∂ θ ) d x = ∫ x − ( ∂ f ( x ; θ ) ∂ θ ) 2 f ( x ; θ ) 2 + ( ∂ 2 f ( x ; θ ) ∂ θ 2 ) f ( x ; θ ) f ( x ; θ ) d x = ∫ x − ( ∂ f ( x ; θ ) ∂ θ ) 2 f ( x ; θ ) 2 d x = − ∫ x ∂ 2 l o g f ( x ; θ ) ∂ θ 2 f ( x ; θ ) d x

Vθ[s(X;θ)]=Eθ[s(X;θ)2]−Eθ[s(X;θ)]2=Eθ[s(X;θ)2]=∫x∂logf(x;θ)∂θ∂logf(x;θ)∂θf(x;θ)dx∫x∂2logf(x;θ)∂θ2f(x;θ)dx=∫x∂∂θ(1f(x;θ)∂f(x;θ)∂θ)dx=∫x−(∂f(x;θ)∂θ)2f(x;θ)2+(∂2f(x;θ)∂θ2)f(x;θ)f(x;θ)dx=∫x−(∂f(x;θ)∂θ)2f(x;θ)2dx=−∫x∂2logf(x;θ)∂θ2f(x;θ)dx" role="presentation" style="position: relative;">Vθ[s(X;θ)]∫x∂2logf(x;θ)∂θ2f(x;θ)dx=Eθ[s(X;θ)2]−Eθ[s(X;θ)]2=Eθ[s(X;θ)2]=∫x∂logf(x;θ)∂θ∂logf(x;θ)∂θf(x;θ)dx=∫x∂∂θ(1f(x;θ)∂f(x;θ)∂θ)dx=∫x−(∂f(x;θ)∂θ)2f(x;θ)2+(∂2f(x;θ)∂θ2)f(x;θ)f(x;θ)dx=∫x−(∂f(x;θ)∂θ)2f(x;θ)2dx=−∫x∂2logf(x;θ)∂θ2f(x;θ)dx$$\begin{array}{rl}{\mathbb{V}}_{\theta }[s(X;\theta )]& =E_{\theta }[s(X;\theta {)}^2]-E_{\theta }[s(X;\theta ){]}^2\\ & =E_{\theta }[s(X;\theta {)}^2]\\ & ={\int }_x\frac{\mathrm{\partial }logf(x;\theta )}{\mathrm{\partial }\theta }\frac{\mathrm{\partial }logf(x;\theta )}{\mathrm{\partial }\theta }f(x;\theta )dx\\ {\int }_x\frac{{\mathrm{\partial }}^{2}logf(x;\theta )}{\mathrm{\partial }{\theta }^2}f(x;\theta )dx& ={\int }_x\frac{\mathrm{\partial }}{\mathrm{\partial }\theta }(\frac{1}{f(x;\theta )}\frac{\mathrm{\partial }f(x;\theta )}{\mathrm{\partial }\theta })dx\\ & ={\int }_x-\frac{(\frac{\mathrm{\partial }f(x;\theta )}{\mathrm{\partial }\theta }{)}^2}{f(x;\theta {)}^2}+\frac{(\frac{{\mathrm{\partial }}^{2}f(x;\theta )}{\mathrm{\partial }{\theta }^2})}{f(x;\theta )}f(x;\theta )dx\\ & ={\int }_x-\frac{(\frac{\mathrm{\partial }f(x;\theta )}{\mathrm{\partial }\theta }{)}^2}{f(x;\theta {)}^2}dx\\ & =-{\int }_x\frac{{\mathrm{\partial }}^{2}logf(x;\theta )}{\mathrm{\partial }{\theta }^2}f(x;\theta )dx\end{array}$$

Vθ​[s(X;θ)]∫x​∂θ2∂2logf(x;θ)​f(x;θ)dx​=Eθ​[s(X;θ)2]−Eθ​[s(X;θ)]2=Eθ​[s(X;θ)2]=∫x​∂θ∂logf(x;θ)​∂θ∂logf(x;θ)​f(x;θ)dx=∫x​∂θ∂​(f(x;θ)1​∂θ∂f(x;θ)​)dx=∫x​−f(x;θ)2(∂θ∂f(x;θ)​)2​+f(x;θ)(∂θ2∂2f(x;θ)​)​f(x;θ)dx=∫x​−f(x;θ)2(∂θ∂f(x;θ)​)2​dx=−∫x​∂θ2∂2logf(x;θ)​f(x;θ)dx​

渐进正态性

极大似然估计具有渐进正态性
$$\frac{{\hat{\theta }}_n-\theta }{se}\to N(0,1)$$
其中，$se\approx \sqrt{\frac{1}{I_n(\theta )}}\approx \sqrt{\frac{1}{I_n(\hat{\theta })}}$
证明从略，资料比较多。

由此可以构建估计的置信区间。

Bernoulli分布的最大似然估计及其方差

设$X_1,\cdots ,X_n\sim Bernoulli(p)$,则其似然函数是$$L(p)=\prod _{i=1}^n{p}^{X_i}(1-p{)}^{1-X_i}$$$$logL(p)=\sum _i^n{X}_{i}logp+(1-X_i)log(1-p)$$
最大化对数似然，就得到：
d d x l o g L ( p ) = 0 ∑ i n X i 1 p − ( 1 − X i ) 1 1 − p = 0 p = 1 n ∑ i = 1 n X i

​dxd​logL(p)=0i∑n​Xi​p1​−(1−Xi​)1−p1​=0p=n1​i=1∑n​Xi​​
其记分函数是：
$$\frac{\partial logL(p)}{\partial p}=\frac{X}{p}-\frac{1-X}{1-p}$$
$$\begin{gathered} I(p)=-E_{\theta }[\frac{d(\frac{X}{p}-\frac{1-X}{1-p})}{dp}]=\frac{1}{1-p}+\frac{1}{p} \\ =\frac{1}{p(1-p)} \end{gathered}$$
$I_n(p)=nI(p)$，估计的方差$$V(p)=np(1-p)\approx n\hat{p}(1-\hat{p})$$