Sign in to save

Bookmark this page so you can find it later.

Sign in to save

Bookmark this page so you can find it later.

Mathematical Statistics Estimators and MLE cheat sheet - grade college

Click image to open full size

Statistics Grade college

Mathematical Statistics Estimators and MLE Cheat Sheet

A printable reference covering estimators, bias, variance, consistency, efficiency, likelihood functions, log-likelihoods, and maximum likelihood estimation for college.

TopicsStatisticsData AnalysisProbabilityFunctionsDerivativesStudy Guide
Download PNG

Related Tools

Related Labs

Related Worksheets

Related Infographics

Study as Flashcards

Mathematical statistics uses estimators to turn sample data into informed guesses about unknown population parameters. This cheat sheet summarizes the main properties used to judge estimators, including bias, variance, mean squared error, consistency, and efficiency. It is useful for comparing methods and understanding why some estimators are preferred in repeated sampling.

Maximum likelihood estimation chooses the parameter value that makes the observed data most probable under a statistical model. The core workflow is to write the likelihood L(θ)L(\theta), take the log-likelihood ℓ(θ)\ell(\theta), differentiate, and solve for the maximizing parameter. Important checks include parameter constraints, second derivative tests, and whether the likelihood maximum occurs at a boundary.

Key Facts

  • An estimator θ^\hat{\theta} is a statistic used to estimate an unknown parameter θ\theta from sample data.
  • The bias of an estimator is Bias⁡(θ^)=E[θ^]−θ\operatorname{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta, so the estimator is unbiased when E[θ^]=θE[\hat{\theta}] = \theta.
  • The variance of an estimator is Var⁡(θ^)=E[(θ^−E[θ^])2]\operatorname{Var}(\hat{\theta}) = E[(\hat{\theta} - E[\hat{\theta}])^2], which measures sampling variability.
  • The mean squared error is MSE⁡(θ^)=E[(θ^−θ)2]=Var⁡(θ^)+Bias⁡(θ^)2\operatorname{MSE}(\hat{\theta}) = E[(\hat{\theta} - \theta)^2] = \operatorname{Var}(\hat{\theta}) + \operatorname{Bias}(\hat{\theta})^2.
  • An estimator θ^n\hat{\theta}_n is consistent for θ\theta if θ^n→pθ\hat{\theta}_n \xrightarrow{p} \theta as n→∞n \to \infty.
  • For independent observations X1,…,XnX_1,\dots,X_n, the likelihood is L(θ)=∏i=1nf(xi∣θ)L(\theta) = \prod_{i=1}^{n} f(x_i \mid \theta).
  • The log-likelihood is ℓ(θ)=log⁡L(θ)=∑i=1nlog⁡f(xi∣θ)\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \log f(x_i \mid \theta), and it has the same maximizer as L(θ)L(\theta).
  • A maximum likelihood estimator satisfies ddθℓ(θ)=0\frac{d}{d\theta}\ell(\theta)=0 at an interior maximum and should also satisfy model constraints such as θ>0\theta > 0.

Vocabulary

Estimator
A statistic θ^\hat{\theta} computed from sample data to estimate an unknown population parameter θ\theta.
Estimate
The numerical value of an estimator after actual sample data have been substituted.
Bias
Bias is the difference E[θ^]−θE[\hat{\theta}] - \theta between an estimator's expected value and the true parameter.
Consistency
Consistency means an estimator θ^n\hat{\theta}_n gets arbitrarily close to θ\theta in probability as nn becomes large.
Likelihood
The likelihood L(θ)L(\theta) treats the observed data as fixed and views the parameter θ\theta as the quantity being compared.
Maximum Likelihood Estimator
The maximum likelihood estimator is the parameter value θ^\hat{\theta} that maximizes L(θ)L(\theta) or equivalently ℓ(θ)\ell(\theta).

Common Mistakes to Avoid

  • Confusing an estimator with an estimate is wrong because θ^\hat{\theta} is a random variable before data are observed, while the estimate is one realized number.
  • Assuming unbiased means best is wrong because an estimator with small bias can have much smaller variance and lower MSE⁡(θ^)\operatorname{MSE}(\hat{\theta}).
  • Maximizing ℓ(θ)\ell(\theta) without checking parameter constraints is wrong because the solution must lie in the allowed parameter space, such as θ>0\theta > 0 or 0<p<10 < p < 1.
  • Forgetting the product in the likelihood for independent samples is wrong because L(θ)=∏i=1nf(xi∣θ)L(\theta)=\prod_{i=1}^{n} f(x_i\mid\theta), not a sum of densities or probabilities.
  • Stopping after solving ℓ′(θ)=0\ell'(\theta)=0 is wrong because critical points can be minima, saddle points, or invalid boundary points rather than maxima.

Practice Questions

  1. 1 Let X1,…,XnX_1,\dots,X_n be a random sample with E[Xi]=μE[X_i]=\mu and Var⁡(Xi)=σ2\operatorname{Var}(X_i)=\sigma^2. Show that Xˉ=1n∑i=1nXi\bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_i is unbiased for μ\mu and find Var⁡(Xˉ)\operatorname{Var}(\bar{X}).
  2. 2 For data x1,…,xnx_1,\dots,x_n from a Bernoulli distribution with parameter pp, write L(p)L(p) and find the maximum likelihood estimator p^\hat{p}.
  3. 3 Suppose X1,…,XnX_1,\dots,X_n are exponential with density f(x∣λ)=λe−λxf(x\mid\lambda)=\lambda e^{-\lambda x} for x≥0x\ge 0 and λ>0\lambda>0. Find the log-likelihood ℓ(λ)\ell(\lambda) and the MLE λ^\hat{\lambda}.
  4. 4 Explain why a biased estimator might still be preferred over an unbiased estimator when comparing MSE⁡(θ^)\operatorname{MSE}(\hat{\theta}).

Understanding Mathematical Statistics Estimators and MLE

A useful way to understand an estimator is to imagine repeating the same study many times. Each random sample gives a slightly different estimate. The collection of all those possible estimates has its own distribution, called the sampling distribution.

Its center shows systematic error, while its spread shows random error. A method can be unbiased yet still give very unstable results in small samples. Another method can miss the target by a little on average but stay tightly grouped around it.

This is why mean squared error is often more useful than bias alone. It combines both kinds of error into one measure of typical squared distance from the true value.

Sample size changes the picture. With more observations, random noise usually averages out, so a sensible estimator becomes more stable. Consistency describes this long run behavior, not a promise that every small sample works well.

A consistent estimator may be quite inaccurate when only a few data values are available. Students should distinguish a parameter from an estimate. The parameter is fixed for the population, even though it is unknown.

The estimate changes from sample to sample. This distinction matters in polling, quality control, medical studies, and sports data.

A poll percentage is not the exact public opinion level. It is one estimate produced by one sample.

Maximum likelihood depends on choosing a probability model before calculating anything. For example, if data are modeled as independent coin flips with one unknown chance of heads, the likelihood rewards values that fit the observed number of heads. If a sample has many heads, larger values of that chance fit better than smaller ones.

The likelihood is not the probability that a parameter is true. In frequentist statistics, the parameter is treated as fixed. Likelihood compares how well different fixed parameter values explain the data that were actually observed.

Multiplying many probabilities can create extremely small numbers, so the log form is used for convenience and numerical safety. The location of the best value stays unchanged because the logarithm preserves order.

Finding a zero derivative is only one step in maximum likelihood work. The candidate may be a minimum, it may fall outside the allowed parameter range, or the best value may lie at an endpoint. A variance estimate, for instance, cannot be negative.

Some models can produce boundary estimates when data contain no observed events or no failures. It is equally important to check whether the model assumptions are believable. Independence can fail for measurements from the same family, classroom, or machine.

A normal model can fit poorly when values are strongly skewed or contain outliers. Maximum likelihood can be mathematically correct for the chosen model while giving a misleading answer if that model does not match the data. Good statistical work includes calculation, assumption checks, and a clear statement of uncertainty.