Maximum likelihood estimation, or MLE, is a method for estimating an unknown parameter from observed data. It asks which parameter value would make the data we actually saw most probable. This idea matters because it is used throughout statistics, machine learning, physics, biology, and economics.
The result is called the maximum likelihood estimate and is usually written as θ̂.
Understanding Statistics: Maximum Likelihood Estimation
A likelihood is built from a model for how data could arise. Suppose a factory fills bottles with a target volume, but the true average fill is unknown. A normal distribution model may describe the small random differences between bottles.
Each possible average gives a different assessment of the observed measurements. Values near the middle of the measurements usually fit better than values far away. The calculation compares many candidate averages using the same data.
The best fitting candidate becomes the estimate. This does not prove that the estimate is the true value. It identifies the value that fits best within the chosen model.
Multiplying probabilities can create extremely tiny numbers when a dataset is large. Logarithms solve this practical problem. A logarithm turns a product into a sum, so a computer can add manageable values rather than multiply many small ones.
Because the logarithm increases whenever its input increases, the parameter that gives the largest likelihood gives the largest log likelihood too. In simple cases, students can find the maximum by using calculus.
They take a derivative of the log likelihood, set it equal to zero, then check that the result is a maximum. In more complex models, software searches for the highest point step by step.
The model assumptions matter as much as the optimization. A common assumption says observations are independent. This means one result does not change the chance of another result once the parameter is known.
Repeated coin flips can often be treated this way. Daily temperatures usually cannot, because today is related to yesterday. Another assumption concerns the distribution shape.
Using a normal distribution for strongly skewed waiting times can give misleading estimates. Outliers can have a large effect, especially when estimating a normal mean and spread. Before trusting an estimate, inspect a graph of the data, consider how it was collected, and decide whether the model has a sensible connection to the real process.
Students meet maximum likelihood ideas whenever data are used to fit a line, classify an image, estimate a disease rate, or calibrate a measuring device. Linear regression can be understood as a likelihood method when measurement errors follow a normal distribution. Many machine learning models adjust their internal parameters to make training examples more likely under the model.
A high likelihood alone does not guarantee useful predictions. A model with too many adjustable parameters may fit random noise in its training data. Testing on new data helps reveal this problem.
It is important to separate uncertainty from the single best estimate. More data usually narrow the uncertainty, while biased sampling can still produce a precise estimate that is wrong for the wider population.
Key Facts
- Likelihood means treating the observed data as fixed and the parameter as the variable.
- For independent data, L(θ) = P(x1 | θ)P(x2 | θ)...P(xn | θ).
- The maximum likelihood estimate is θ̂ = arg max L(θ).
- It is often easier to maximize the log-likelihood: ℓ(θ) = ln L(θ).
- For independent data, ℓ(θ) = ln P(x1 | θ) + ln P(x2 | θ) + ... + ln P(xn | θ).
- For a coin with h heads in n flips, the MLE for the probability of heads is p̂ = h/n.
Vocabulary
- Parameter
- A parameter is an unknown number in a statistical model, such as a mean, probability, or rate.
- Likelihood function
- A likelihood function gives how plausible different parameter values are after the data have been observed.
- Maximum likelihood estimate
- A maximum likelihood estimate is the parameter value that gives the largest likelihood for the observed data.
- Log-likelihood
- The log-likelihood is the natural logarithm of the likelihood function, often used because sums are easier to work with than products.
- Independent observations
- Independent observations are data points whose probabilities can be multiplied because one observation does not affect another.
Common Mistakes to Avoid
- Confusing likelihood with probability is wrong because likelihood varies the parameter after the data are fixed, while probability usually varies possible data for a fixed parameter.
- Forgetting to multiply probabilities for independent observations is wrong because the likelihood of the whole data set must combine all observations, not just one data point.
- Maximizing the likelihood instead of the log-likelihood by hand can lead to unnecessary algebra errors because the log-likelihood has the same maximum but is usually simpler.
- Assuming the highest point of a plotted curve is always meaningful without checking the model is wrong because MLE depends on choosing an appropriate statistical model for the data.
Practice Questions
- 1 A coin is flipped 20 times and lands heads 13 times. Using maximum likelihood estimation, what is p̂ for the probability of heads?
- 2 A biased die has probability θ of rolling a 6. In 50 rolls, a 6 appears 8 times. Write the likelihood L(θ) up to a constant factor and find the MLE θ̂.
- 3 Two models give different likelihood curves for the same data. Model A has a sharp high peak, while Model B has a lower but wider peak. Explain what the peak location tells you and why the width of the curve matters for uncertainty.