Backend & Data
Redis Streams: a to-do list many workers can share
Imagine a busy deli counter. Customers take numbered tickets; the numbers only ever go up; and several servers behind the counter each call "next!" — but no two ever serve the same ticket. That deli counter is, more or less, a Redis Stream.
What a stream is
A Redis Stream is an append-only log. Every item you add gets an ever-increasing ID and takes its place in order. Nothing is overwritten and nothing is lost when it's read — instead, each reader keeps track of where it has got to. That one design choice unlocks everything else.
Consumer groups: sharing the work
A consumer group is how several workers split one stream. The group makes a promise: each entry is delivered to exactly one worker in the group. Need more throughput? Add another worker and the group spreads the load automatically.
There's a safety net too. A worker must acknowledge (ACK) an entry once it's finished. If the worker crashes mid-task without ACKing, the entry is still pending and can be handed to someone else — so work isn't silently dropped. This is "at-least-once" delivery.
Producers append to the log. A consumer group hands each entry to one worker. Workers ACK when done — and anything left unacknowledged can be retried.
Why it fits real-time systems
- A durable buffer between a fast producer (say, a per-minute market-data feed) and slower consumers that crunch it.
- Replay. Because entries stay in the log, you can re-read history from any point — invaluable for testing.
- Many readers, one stream. Independent groups can each read the same data for different purposes without interfering.
It's a small primitive with a big payoff: the moment your "just process this queue" turns into "process this queue reliably, across a fleet of workers, without losing anything," a stream with consumer groups is often the cleanest answer.