Kafka is a distributed event streaming platform used to move, store, and process high volume data in real time. This cheat sheet summarizes the architecture, consumer behavior, reliability settings, and Kafka Streams DSL patterns that college students need when designing streaming systems. It helps connect core theory, such as partitioning and fault tolerance, with practical API concepts used in production applications.
The most important ideas are that topics are split into partitions, records are ordered only within a partition, and consumers track progress using offsets. Reliability depends on replication, acknowledgments, idempotence, transactions, and careful offset commits. Kafka Streams adds abstractions such as KStream, KTable, joins, windows, and state stores for building stream processing applications directly on Kafka.
Key Facts
- A Kafka topic is a named stream of records, and each topic is divided into partitions for scalability and parallel processing.
- Kafka preserves record order only within a single partition, not across all partitions in a topic.
- A record key usually determines the partition, so records with the same key are commonly routed to the same partition.
- A consumer group shares work across consumers, and each partition is assigned to at most one consumer in the same group at a time.
- An offset is the position of a record in a partition, and committing an offset records how far a consumer has processed.
- Replication factor is the number of copies of each partition, and min.insync.replicas sets how many replicas must acknowledge writes for stronger durability.
- Producer setting acks=all requires acknowledgments from all in-sync replicas, which improves durability compared with acks=0 or acks=1.
- Kafka Streams represents an event stream with KStream, a changelog-style table with KTable, and time-based grouping with windowedBy().
Vocabulary
- Broker
- A Kafka server that stores topic partitions, serves reads and writes, and participates in cluster coordination.
- Topic
- A named category of records that producers write to and consumers read from.
- Partition
- An ordered, append-only log within a topic that allows Kafka to scale storage and processing.
- Offset
- A numeric position that identifies a record within a specific topic partition.
- Consumer Group
- A set of consumers that coordinate to divide partitions among themselves so messages are processed in parallel.
- State Store
- A local, fault-tolerant storage component used by Kafka Streams to maintain tables, aggregations, joins, and windowed results.
Common Mistakes to Avoid
- Assuming Kafka guarantees global ordering is wrong because ordering is guaranteed only within each partition.
- Using too few partitions limits throughput and consumer parallelism because each partition can be consumed by only one consumer in a group at a time.
- Committing offsets before processing finishes is unsafe because a crash can make Kafka think records were handled when they were not.
- Ignoring record keys can break grouping and joins because related records may be sent to different partitions.
- Treating KStream and KTable as the same abstraction is wrong because KStream models independent events while KTable models the latest value for each key.
Practice Questions
- 1 A topic has 12 partitions and one consumer group has 4 active consumers. If partitions are balanced evenly, how many partitions does each consumer read?
- 2 A topic has replication factor 3 and min.insync.replicas=2. If the producer uses acks=all, how many in-sync replicas must acknowledge the write for it to succeed?
- 3 A consumer processes records at offsets 20 through 29 and then commits offset 30. What does the committed offset 30 mean?
- 4 Explain why choosing a stable key is important when using Kafka Streams for joins or aggregations.