Why Real-Time AI at Scale Is So Hard
ScyllaDB, Thursday, October 29th, 2026
ScyllaDB engineers explain why real-time AI pipelines fail at scale: tail latency, stale features and retraining contention.
Based on a discussion with Tim Koopmans, who built an ML-based trading app, the post lists failure patterns in real-time AI.
Near 740K operations per second, P99 latency hit 3 seconds because feature reads queued behind writes in Postgres, and tail latency was an architectural property rather than a bug.
Stale features and embeddings caused accuracy drops that offline metrics missed, and retraining caused contention and a warm-up window during model swaps, all of which feed a doom loop.
Advice includes monitoring freshness and backlog, separating read and write paths, and using blue-green rollouts, and ScyllaDB's lock-free writes and workload prioritization are promoted.