The systems, abstractions, and operational layers that make AI models usable at scale, from compute and caching to routing, governance, agent hosting, and credential management.

LLM inference covers how language models generate tokens from a prompt — spanning hardware constraints, serving architecture, caching strategies, quantization, routing, and cost — and has become its own engineering discipline as scale and cost pressures intensify.

The infrastructure, utilities, and integration layers built around large language models, spanning local inference runtimes, context management, MCP servers, knowledge organization, and provider-agnostic design patterns.

Open source spans infrastructure, tooling, security risk, and platform trust — the cited sources collectively show it as a foundation for local AI, developer tooling, and code forges, with its benefits shadowed by real supply-chain and stewardship threats.

The engineering decisions that determine how software behaves under real load, covering durability, observability, testing discipline, performance constraints, and the operational costs of failure.