A search engine turns a few typed words into a ranked list of useful web pages in a fraction of a second. It matters because the web is far too large for people to browse page by page, so software must discover, organize, and evaluate information automatically. Modern search combines computer science ideas from data structures, algorithms, databases, networking, and artificial intelligence.
The visible results page is only the final step of a long pipeline.
Key Facts
- Crawling discovers pages by following links and reading URLs from sources such as sitemaps.
- Indexing stores words, page locations, metadata, and links so pages can be found quickly later.
- An inverted index maps each term to the documents that contain it, such as apple -> Doc 2, Doc 8, Doc 19.
- TF-IDF gives higher weight to terms that are frequent in one document but rare across many documents: TF-IDF = TF x IDF.
- A simple ranking model can be written as Score = relevance + authority + freshness + usability.
- Query processing may include spelling correction, tokenization, synonym matching, and intent detection before ranking begins.
Vocabulary
- Crawler
- A crawler is a program that automatically visits web pages and follows links to discover more pages.
- Index
- An index is a searchable data structure that stores information about web pages so results can be retrieved quickly.
- Inverted index
- An inverted index maps each word or token to the list of documents where that word appears.
- Ranking algorithm
- A ranking algorithm orders search results by estimating which pages are most useful for a query.
- Query
- A query is the text, voice command, or other input that a user gives to a search engine.
Common Mistakes to Avoid
- Thinking the search engine searches the whole web live for every query. This is wrong because most results come from a prebuilt index, which is much faster than visiting pages in real time.
- Confusing crawling with indexing. Crawling discovers and downloads pages, while indexing analyzes and stores their contents for later search.
- Assuming the page with the most repeated keywords should always rank first. This is wrong because search engines also consider relevance, link authority, quality, location, freshness, and user intent.
- Ignoring how wording changes results. Small changes such as adding a specific term, using quotes, or including a location can change how the query is interpreted and ranked.
Practice Questions
- 1 A crawler visits 120 pages per minute. If 15 percent of the pages are duplicates and are not indexed, how many pages are added to the index in 10 minutes?
- 2 A search engine scores a page using Score = 0.5R + 0.3A + 0.2F, where R is relevance, A is authority, and F is freshness. If R = 80, A = 60, and F = 50, what is the page score?
- 3 Two pages both contain the exact query phrase. Page A is old but has many trusted sites linking to it, while Page B is new and mobile friendly but has few links. Explain why a search engine might rank either page higher depending on the query intent.