Asymptotics of MLE
Published:
This post summarizes asymptotic properties of MLEs.
Suppose we draw independent X1,…,Xn∼p(X;θ), and obtain ˆθMLE.
Consistency
We can treat MLE as empirical risk minimization where the empirical risk is defined as $$\hat{R}n(\hat{\theta},\theta)=\frac{1}{n}\sum\limits{i=1}^n\log\frac{p(X_i;\theta)}{p(X_i;\hat{\theta})}.$$ Thus, $\hat{\theta}{MLE}=\mathop{\arg\min}{\hat{\theta}}\hat{R}n(\hat{\theta},\theta).Notethatthetrueriskisgivenby$R(\hat{\theta},\theta)=\mathbb{E}{p(X;\theta)}[\hat{R}_n(\hat{\theta},\theta)]=KL(p(X;\theta)\|p(X;\hat{\theta})),$$ which is positive if p(X;θ)≠p(X;ˆθ) almost surely.