Production systems

Production systems are where abstractions meet reality. The gap between what software does in development and what it does under live load is where most interesting engineering problems live, and the sources here circle that gap from several angles.

Durability is one axis. Temporal persists workflow state at every step so distributed applications recover from failures automatically. Jack Vanlightly’s taxonomy maps durable execution into three forms — stateless functions, sessions, and actors — and shows how Temporal, Restate, DBOS, and Resonate each implement them. Depot’s CI orchestrator applies the same principle differently, using AWS Lambda durable functions to run a stateful, checkpointed workflow scheduler without keeping a long-lived process alive.

Failure modes in production are rarely dramatic. The GitHub merge queue bug described by Trunk silently deleted thousands of lines from main branches because temp branches were built off the wrong base commit. The Unwritten Laws of Software Engineering draws the same lesson in principle: roll back before debugging, and treat every external dependency as a future outage.

Observability is the mechanism for understanding what actually happened. Distributed traces let engineers read unfamiliar codebases by examining span anatomy and critical-path analysis rather than source code alone. LangChain’s Harrison Chase extends this: traces alone don’t improve agentic systems; feedback signals attached to traces are what turn observability into a learning loop.

Testing discipline connects staging and production concerns. Currents provides a decision framework for splitting Playwright tests between environments, including the operational costs of testing in production. Emphere takes a harder line with their container security tool, using red runs that prove the system fails loudly when it overclaims certainty.

Performance has its own traps. Colin Breck identifies three constraints — attention thresholds, discrete capacity increments, and pipeline backpressure — that explain why order-of-magnitude improvements often fail to change outcomes. Linear’s architecture illustrates the other side: local-first IndexedDB sync, optimistic updates, and aggressive code splitting compound into perceived near-instant performance. LLM inference has its own cost dynamics; KV caching treated as a persistent shared asset rather than a per-request computation can cut prefill costs by up to 20x.

Configuration correctness is an underrated production concern. YAML’s Norway problem — where the country code NO parses as false — persists in popular libraries a decade after the spec fixed it, illustrating how configuration formats carry silent failure modes into production deployments.