Lambda architecture.
Running a batch layer (accurate, periodic) and a speed layer (real-time, approximate) side by side and merging their results at query time.
the two-minute refresher
- Batch layer: recomputes from all history, the source of truth for accuracy.
- Speed layer: covers the gap since the last batch, discarded when the batch catches up.
- Serving layer: batch view + real-time view at query time.
- The big problem: two codebases (Spark and Flink) that must produce the same answer.