Sign in to save

Bookmark this page so you can find it later.

Sign in to save

Bookmark this page so you can find it later.

Mathematical Statistics Estimators and MLE cheat sheet - grade college

Click image to open full size

Statistics Grade college

Mathematical Statistics Estimators and MLE Cheat Sheet

A printable reference covering estimators, bias, variance, consistency, efficiency, likelihood functions, log-likelihoods, and maximum likelihood estimation for college.

TopicsStatisticsData AnalysisProbabilityFunctionsDerivativesStudy Guide
Download PNG

Related Tools

Related Labs

Related Worksheets

Related Infographics

Study as Flashcards

Mathematical statistics uses estimators to turn sample data into informed guesses about unknown population parameters. This cheat sheet summarizes the main properties used to judge estimators, including bias, variance, mean squared error, consistency, and efficiency. It is useful for comparing methods and understanding why some estimators are preferred in repeated sampling.

Maximum likelihood estimation chooses the parameter value that makes the observed data most probable under a statistical model. The core workflow is to write the likelihood L(θ)L(\theta), take the log-likelihood ℓ(θ)\ell(\theta), differentiate, and solve for the maximizing parameter. Important checks include parameter constraints, second derivative tests, and whether the likelihood maximum occurs at a boundary.

Key Facts

  • An estimator θ^\hat{\theta} is a statistic used to estimate an unknown parameter θ\theta from sample data.
  • The bias of an estimator is Bias⁡(θ^)=E[θ^]−θ\operatorname{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta, so the estimator is unbiased when E[θ^]=θE[\hat{\theta}] = \theta.
  • The variance of an estimator is Var⁡(θ^)=E[(θ^−E[θ^])2]\operatorname{Var}(\hat{\theta}) = E[(\hat{\theta} - E[\hat{\theta}])^2], which measures sampling variability.
  • The mean squared error is MSE⁡(θ^)=E[(θ^−θ)2]=Var⁡(θ^)+Bias⁡(θ^)2\operatorname{MSE}(\hat{\theta}) = E[(\hat{\theta} - \theta)^2] = \operatorname{Var}(\hat{\theta}) + \operatorname{Bias}(\hat{\theta})^2.
  • An estimator θ^n\hat{\theta}_n is consistent for θ\theta if θ^n→pθ\hat{\theta}_n \xrightarrow{p} \theta as n→∞n \to \infty.
  • For independent observations X1,…,XnX_1,\dots,X_n, the likelihood is L(θ)=∏i=1nf(xi∣θ)L(\theta) = \prod_{i=1}^{n} f(x_i \mid \theta).
  • The log-likelihood is ℓ(θ)=log⁡L(θ)=∑i=1nlog⁡f(xi∣θ)\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \log f(x_i \mid \theta), and it has the same maximizer as L(θ)L(\theta).
  • A maximum likelihood estimator satisfies ddθℓ(θ)=0\frac{d}{d\theta}\ell(\theta)=0 at an interior maximum and should also satisfy model constraints such as θ>0\theta > 0.

Vocabulary

Estimator
A statistic θ^\hat{\theta} computed from sample data to estimate an unknown population parameter θ\theta.
Estimate
The numerical value of an estimator after actual sample data have been substituted.
Bias
Bias is the difference E[θ^]−θE[\hat{\theta}] - \theta between an estimator's expected value and the true parameter.
Consistency
Consistency means an estimator θ^n\hat{\theta}_n gets arbitrarily close to θ\theta in probability as nn becomes large.
Likelihood
The likelihood L(θ)L(\theta) treats the observed data as fixed and views the parameter θ\theta as the quantity being compared.
Maximum Likelihood Estimator
The maximum likelihood estimator is the parameter value θ^\hat{\theta} that maximizes L(θ)L(\theta) or equivalently ℓ(θ)\ell(\theta).

Common Mistakes to Avoid

  • Confusing an estimator with an estimate is wrong because θ^\hat{\theta} is a random variable before data are observed, while the estimate is one realized number.
  • Assuming unbiased means best is wrong because an estimator with small bias can have much smaller variance and lower MSE⁡(θ^)\operatorname{MSE}(\hat{\theta}).
  • Maximizing ℓ(θ)\ell(\theta) without checking parameter constraints is wrong because the solution must lie in the allowed parameter space, such as θ>0\theta > 0 or 0<p<10 < p < 1.
  • Forgetting the product in the likelihood for independent samples is wrong because L(θ)=∏i=1nf(xi∣θ)L(\theta)=\prod_{i=1}^{n} f(x_i\mid\theta), not a sum of densities or probabilities.
  • Stopping after solving ℓ′(θ)=0\ell'(\theta)=0 is wrong because critical points can be minima, saddle points, or invalid boundary points rather than maxima.

Practice Questions

  1. 1 Let X1,…,XnX_1,\dots,X_n be a random sample with E[Xi]=μE[X_i]=\mu and Var⁡(Xi)=σ2\operatorname{Var}(X_i)=\sigma^2. Show that Xˉ=1n∑i=1nXi\bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_i is unbiased for μ\mu and find Var⁡(Xˉ)\operatorname{Var}(\bar{X}).
  2. 2 For data x1,…,xnx_1,\dots,x_n from a Bernoulli distribution with parameter pp, write L(p)L(p) and find the maximum likelihood estimator p^\hat{p}.
  3. 3 Suppose X1,…,XnX_1,\dots,X_n are exponential with density f(x∣λ)=λe−λxf(x\mid\lambda)=\lambda e^{-\lambda x} for x≥0x\ge 0 and λ>0\lambda>0. Find the log-likelihood ℓ(λ)\ell(\lambda) and the MLE λ^\hat{\lambda}.
  4. 4 Explain why a biased estimator might still be preferred over an unbiased estimator when comparing MSE⁡(θ^)\operatorname{MSE}(\hat{\theta}).