Kafka is a distributed event streaming platform used to move, store, and process high volume data in real time. This cheat sheet summarizes the architecture, consumer behavior, reliability settings, and Kafka Streams DSL patterns that college students need when designing streaming systems. It helps connect core theory, such as partitioning and fault tolerance, with practical API concepts used in production applications.
The most important ideas are that topics are split into partitions, records are ordered only within a partition, and consumers track progress using offsets. Reliability depends on replication, acknowledgments, idempotence, transactions, and careful offset commits. Kafka Streams adds abstractions such as KStream, KTable, joins, windows, and state stores for building stream processing applications directly on Kafka.
Key Facts
- A Kafka topic is a named stream of records, and each topic is divided into partitions for scalability and parallel processing.
- Kafka preserves record order only within a single partition, not across all partitions in a topic.
- A record key usually determines the partition, so records with the same key are commonly routed to the same partition.
- A consumer group shares work across consumers, and each partition is assigned to at most one consumer in the same group at a time.
- An offset is the position of a record in a partition, and committing an offset records how far a consumer has processed.
- Replication factor is the number of copies of each partition, and min.insync.replicas sets how many replicas must acknowledge writes for stronger durability.
- Producer setting acks=all requires acknowledgments from all in-sync replicas, which improves durability compared with acks=0 or acks=1.
- Kafka Streams represents an event stream with KStream, a changelog-style table with KTable, and time-based grouping with windowedBy().
Vocabulary
- Broker
- A Kafka server that stores topic partitions, serves reads and writes, and participates in cluster coordination.
- Topic
- A named category of records that producers write to and consumers read from.
- Partition
- An ordered, append-only log within a topic that allows Kafka to scale storage and processing.
- Offset
- A numeric position that identifies a record within a specific topic partition.
- Consumer Group
- A set of consumers that coordinate to divide partitions among themselves so messages are processed in parallel.
- State Store
- A local, fault-tolerant storage component used by Kafka Streams to maintain tables, aggregations, joins, and windowed results.
Common Mistakes to Avoid
- Assuming Kafka guarantees global ordering is wrong because ordering is guaranteed only within each partition.
- Using too few partitions limits throughput and consumer parallelism because each partition can be consumed by only one consumer in a group at a time.
- Committing offsets before processing finishes is unsafe because a crash can make Kafka think records were handled when they were not.
- Ignoring record keys can break grouping and joins because related records may be sent to different partitions.
- Treating KStream and KTable as the same abstraction is wrong because KStream models independent events while KTable models the latest value for each key.
Practice Questions
- 1 A topic has 12 partitions and one consumer group has 4 active consumers. If partitions are balanced evenly, how many partitions does each consumer read?
- 2 A topic has replication factor 3 and min.insync.replicas=2. If the producer uses acks=all, how many in-sync replicas must acknowledge the write for it to succeed?
- 3 A consumer processes records at offsets 20 through 29 and then commits offset 30. What does the committed offset 30 mean?
- 4 Explain why choosing a stable key is important when using Kafka Streams for joins or aggregations.
Understanding Kafka Streaming Reference
Kafka works like a durable append-only log. Producers add records to the end of a partition, while brokers write them to disk in ordered segments. Old segments can be removed by a retention time or a storage size rule.
This means Kafka is not only a message queue. A new consumer can read earlier records if they are still retained. That ability is useful for rebuilding a search index, training an analytics model, or repairing a database after an application bug.
It also means storage planning matters. A topic receiving large images, logs, or sensor readings can fill disks quickly if retention is chosen carelessly.
Keys have an important design role. A payment identifier, customer identifier, or device identifier can keep related events together. Processing then sees a meaningful local order for that one entity.
A poor key can create hot partitions. For example, using a country as the key may send most records to one partition if most users come from one country. That partition becomes the bottleneck while other consumers wait with little work.
Null keys may be spread across partitions, but they do not preserve grouping for one entity. Students should practice choosing keys by asking which records must be processed in sequence and how evenly the expected traffic will be distributed.
Consumer failures are normal, so applications must expect repeated processing. A consumer may process a record, fail before its progress is committed, then receive that record again after restarting. Code should therefore be idempotent when possible.
An idempotent operation gives the same final result even if it runs more than once. Saving an order status with a stable order identifier is safer than blindly adding a payment amount each time. Consumer group rebalances can cause another pause.
They happen when consumers join, leave, or fail, because partition assignments must change. Long running record handling can interfere with this process. In real systems, teams often move slow external calls out of the consumer path or tune polling settings carefully.
Kafka Streams handles many details of consuming, producing, and tracking local state, but its processing choices still matter. A KStream represents individual facts, such as every card swipe. A KTable represents the latest known value for each key, such as the current account profile.
Updates to a KTable may replace earlier values for that key, and deleted values are represented with tombstone records. Windowed calculations depend on event time, not merely the time a program receives data. Late events can arrive because networks fail, mobile devices reconnect, or clocks differ.
A grace period decides how long the program waits for late data. Joins and aggregations create state stores, which Kafka Streams backs up through changelog topics. Students should inspect input keys, timestamps, window boundaries, output records, and recovery behavior when testing a stream program.