A foundation model is a large AI model trained on huge amounts of data so it can learn patterns that are useful for many tasks. Instead of building a separate model from scratch for every job, engineers can adapt one foundation model for writing, coding, image understanding, translation, and more. This matters because foundation models power many tools students already see, such as chatbots, search assistants, and image generators.
They are called foundation models because many applications can be built on top of them.
Key Facts
- A foundation model is trained on large, varied datasets to learn general patterns before being adapted to specific tasks.
- Training adjusts model parameters to reduce error: loss = prediction error.
- A simple learning update is new weight = old weight - learning rate × gradient.
- More parameters can let a model learn more complex patterns, but size alone does not guarantee accuracy or fairness.
- Fine-tuning adapts a pretrained model using a smaller task-specific dataset.
- Model output is probabilistic: the model often chooses the next token with high estimated probability, such as P(next token | context).
Vocabulary
- Foundation model
- A foundation model is a large AI model trained on broad data so it can be adapted for many different tasks.
- Training data
- Training data is the collection of examples a model uses to learn patterns during training.
- Parameter
- A parameter is a number inside a model that changes during training to improve predictions.
- Fine-tuning
- Fine-tuning is the process of adapting a pretrained model to a more specific task using additional examples.
- Token
- A token is a small unit of text, such as a word part or symbol, that a language model processes.
Common Mistakes to Avoid
- Thinking a foundation model understands like a human is wrong because it learns statistical patterns in data, not personal experience or consciousness.
- Assuming bigger always means better is wrong because data quality, training method, testing, and safety design also strongly affect performance.
- Forgetting to check sources is wrong because foundation models can generate confident answers that are inaccurate or unsupported.
- Believing training data is perfectly neutral is wrong because datasets can contain bias, missing groups, errors, and outdated information.
Practice Questions
- 1 A model is trained on 600 billion tokens. If 15 percent of the tokens come from science and math text, how many science and math tokens are in the training data?
- 2 A smaller AI model has 2 billion parameters, and a larger foundation model has 70 billion parameters. How many times more parameters does the larger model have?
- 3 A school wants to use a foundation model to help students study, but the model sometimes gives incorrect explanations. What steps should the school take before trusting it as a learning tool?