GenAI
Architecture
A comprehensive reference for architecting generative AI systems — from single-call inference to retrieval-augmented pipelines, from single agents to multi-agent orchestration. Covers the full decision space: model serving, RAG, fine-tuning, context engineering, agent design patterns, memory, guardrails, and production-grade agentic system design.
What Is GenAI Architecture
GenAI architecture is the discipline of designing the system around a foundation model — the retrieval, context assembly, tool access, memory, orchestration, guardrails, and observability that turn a raw next-token predictor into a reliable product. The model is the engine; architecture is the chassis, transmission, and steering that determine whether the engine is useful, safe, and cost-effective in production.
Three architectural concerns dominate almost every GenAI system, and this handbook is structured around them: (1) GenAI architecture patterns — how you get the right knowledge and context in front of a model (RAG, fine-tuning, embeddings, caching); (2) agent patterns — how a model reasons, calls tools, and iterates toward a goal; and (3) agentic system design — how you compose many agents, tools, and safety layers into a system that can run with reduced or zero human supervision.
A single (or chained) LLM call with retrieved or provided context, producing one output per input. No autonomous looping, no tool-driven branching. Examples: RAG Q&A, summarization, classification, content generation pipelines.
The model decides its own next action — which tool to call, whether to retrieve more information, when it's done — in a loop, with an explicit stopping condition. Examples: coding agents, research agents, customer-support resolution agents.
Multiple specialized agents (each potentially a different model, prompt, or tool scope) coordinate through an orchestrator, message bus, or shared state to solve tasks no single agent handles well alone.
The GenAI Reference Stack
Nearly every production GenAI system — RAG app, coding agent, support bot — can be decomposed into the same seven layers. Not every system needs every layer fully built out, but knowing the full stack helps you see what you're skipping and why.
Model Layer & Serving Patterns
The model layer is rarely "one model, one endpoint" in production. It's a small distributed system in its own right: routing requests to the right model, falling back on failure, caching repeated prefixes, and batching where latency budgets allow.
A thin service in front of one or more model providers. Normalizes request/response schemas across providers, applies retry/backoff, enforces rate limits and budgets per tenant, and centralizes API-key management. Examples of the pattern: LiteLLM, Portkey, or a custom gateway.
Route "easy" requests to a small, cheap, fast model and escalate to a larger model only when a cheap classifier, confidence score, or the small model's own uncertainty signals it can't handle the task. Cuts cost and latency substantially at scale without much quality loss.
If the primary provider errors, rate-limits, or times out, fail over to a secondary provider or a self-hosted model. Requires prompt templates that are portable across providers and a health-check loop that tracks provider latency/error rates in real time.
Self-hosting (vLLM, TGI, SGLang) buys data residency, fixed infrastructure cost at scale, and custom fine-tunes — at the cost of ops burden (GPU fleet, autoscaling, quantization tuning). API-hosted models buy speed-to-market and access to frontier capability without infra ownership.
| Serving Concern | Technique | Effect |
|---|---|---|
| Latency | Streaming responses, speculative decoding, prompt caching | Lower time-to-first-token and perceived latency |
| Throughput | Continuous batching (vLLM/TGI), KV-cache reuse | Higher requests/sec per GPU |
| Cost | Model cascades, prompt caching, output length limits, smaller distilled models | Lower $ per resolved request |
| Reliability | Multi-provider fallback, circuit breakers, timeout budgets | Graceful degradation instead of hard failure |
| Determinism | Low/zero temperature, seeded sampling, structured output (JSON schema / grammars) | More reproducible behavior for evals and automation |
RAG Architecture Patterns
Retrieval-Augmented Generation grounds model outputs in external, up-to-date, and authoritative data instead of relying solely on parametric (trained-in) knowledge. RAG is the single most common GenAI architecture pattern because it solves the two biggest LLM limitations — stale knowledge and hallucination — without retraining the model.
RAG Pattern Variants
Single retrieval pass → top-k chunks → stuffed into the prompt → one generation call. Fast to build, works for narrow, well-chunked knowledge bases. Fails on multi-hop questions and queries that need synthesis across many documents.
Adds query rewriting/expansion, hybrid (dense + sparse) retrieval, reranking, and metadata filtering before generation. This is the realistic production baseline — naive RAG rarely survives contact with real user queries.
The model itself decides whether to retrieve, what to query for, and whether retrieved context is sufficient — looping retrieval and generation until it has enough grounding. Handles multi-hop and ambiguous queries at the cost of latency and complexity.
Builds a knowledge graph (entities + relationships) from the corpus, often with LLM-assisted extraction, then retrieves via graph traversal and/or community summaries alongside or instead of vector search. Strong for "what connects to what" and global-summary questions that pure chunk retrieval handles poorly.
The model first generates a hypothetical answer to the query, then embeds that hypothetical answer to search the index — closing the semantic gap between short questions and long documents. Useful when queries and documents live in very different linguistic registers.
A lightweight evaluator scores retrieved chunks for relevance; low-relevance results trigger a fallback (query rewrite, web search, or a "not confident" response) instead of feeding weak context straight to generation. Reduces hallucination from bad retrieval.
Chunking Strategy
| Strategy | How It Works | Best For |
|---|---|---|
| Fixed-size | Split every N tokens with overlap (e.g. 512 tokens, 15% overlap) | Simple, homogeneous text; fast baseline |
| Recursive / structure-aware | Split on headings → paragraphs → sentences, respecting document structure | Markdown, HTML, structured docs |
| Semantic chunking | Split where embedding similarity between adjacent sentences drops sharply | Long-form prose where topic boundaries matter |
| Sentence-window | Embed single sentences; retrieve with surrounding sentence window for context | Precise retrieval + readable surrounding context |
| Parent-document / hierarchical | Embed small chunks for retrieval precision; return the larger parent chunk/section for generation | Balancing retrieval precision with generation context |
| Table/code-aware | Keep tables, code blocks, and structured elements intact rather than splitting mid-structure | Technical docs, financial reports, source code |
Retrieval & Reranking
- Dense (embeddings): captures semantic similarity, robust to paraphrasing
- Sparse (BM25 / keyword): captures exact term matches — IDs, codes, rare terms embeddings often miss
- Combine via reciprocal rank fusion (RRF) or weighted score blending
- Retrieve top-50 cheaply with bi-encoder/BM25, then rerank with a cross-encoder for top-5
- Cross-encoders score query+doc jointly — far more accurate than embedding cosine similarity alone
- Cohere Rerank, BGE-reranker, or a fine-tuned cross-encoder are common choices
Fine-Tuning & Adaptation Architecture
Fine-tuning bakes knowledge or behavior into model weights rather than supplying it at inference time. It is the right tool for a narrower set of problems than most teams assume — most "we need to fine-tune" problems are actually RAG or prompt-engineering problems in disguise.
| Need | Right Tool | Why |
|---|---|---|
| Model needs facts it doesn't know / that change often | RAG | Fine-tuning is a poor way to inject frequently-changing factual knowledge — it's slow to update and doesn't reliably "recall" specific facts |
| Model needs to follow a specific output format or tone consistently | Prompting + few-shot, or light fine-tuning | Often solvable with better system prompts and examples before touching weights |
| Model needs a new skill/behavior pattern (e.g. a specific tool-call style, a narrow domain reasoning style) | Fine-tuning (SFT/LoRA) | Behavior distillation across many examples generalizes better via weights than via ever-longer prompts |
| Need to shrink a large model into a small, cheap, fast one for a narrow task | Distillation fine-tuning | Train a small model on the large model's outputs for the specific task distribution |
| Need consistent alignment to human/organizational preferences | RLHF / DPO / preference tuning | Preference optimization shapes judgment and style, not factual recall |
Update all model weights. Highest capability ceiling, highest cost, highest infrastructure requirement (multi-GPU, large memory). Risk of catastrophic forgetting of general capability. Reserved for teams with the data volume and infra to justify it.
Freeze the base model, train small low-rank adapter matrices injected into attention/MLP layers. Trains 0.1–1% of parameters, runs on far less hardware, and adapters can be swapped per task/tenant at inference time without redeploying the base model.
Train on curated (instruction, ideal-response) pairs to shape behavior — format compliance, tool-call style, domain tone. The most common fine-tuning pattern for production GenAI products layered on top of a strong base model.
Train on (chosen, rejected) response pairs to shape judgment — which of two plausible responses is better. Used to reduce specific failure patterns (excessive hedging, refusals, verbosity) once SFT has established base behavior.
Architectural Considerations for Fine-Tuned Models in Production
With LoRA, you can serve many task- or tenant-specific adapters on one base model deployment (e.g. via multi-LoRA serving in vLLM), switching adapters per request instead of hosting N full model copies — a major cost lever for multi-tenant GenAI products.
Fine-tuning without a regression eval suite is how teams silently degrade general capability while fixing one narrow behavior. Maintain a fixed eval set spanning the target skill and general capability, run before/after every training run, and gate promotion on both improving or holding steady.
Most of the engineering effort in a fine-tuning program is the data pipeline: collection, deduplication, quality filtering, PII scrubbing, and versioning of training data — not the training run itself, which is often a few lines of a training framework's API.
Embedding & Vector Store Architecture
Every RAG and semantic-search system rests on an embedding model and a vector index. The choices here — embedding model, index algorithm, and storage engine — drive retrieval quality, latency, and cost far more than most teams expect.
- General-purpose: OpenAI/Voyage/Cohere embedding APIs — strong out of the box, no infra to manage
- Open-weight: BGE, E5, GTE families — self-hostable, tunable, no per-call cost at scale
- Domain-tuned: fine-tuned or contrastively-trained embeddings for legal, medical, or code domains outperform general models on in-domain retrieval
- Multi-vector: ColBERT-style late-interaction models trade index size for materially better retrieval precision
- HNSW — graph-based approximate nearest neighbor; the default for most production vector DBs. Fast queries, higher memory footprint
- IVF (+PQ) — cluster-then-search; lower memory via product quantization, some recall trade-off
- Exact / brute-force — fine for <100K vectors, guarantees perfect recall
- DiskANN — disk-resident graph index for billion-scale corpora that don't fit in memory
| Store Type | Examples | Choose When |
|---|---|---|
| Dedicated vector DB | Pinecone, Weaviate, Qdrant, Milvus | Vector search is a primary workload, need managed scaling and hybrid search built-in |
| Postgres extension | pgvector | Already running Postgres; want vectors alongside relational data in one system; moderate scale |
| Search-engine native | Elasticsearch/OpenSearch with vector fields | Already running a search stack; need tight hybrid lexical+vector search |
| In-memory / embedded | FAISS, Chroma (local mode) | Prototyping, small corpora, offline batch pipelines |
| Graph database | Neo4j, graph + vector hybrids | GraphRAG, relationship-heavy retrieval alongside similarity search |
Prompt & Context Architecture
Context engineering — deciding exactly what tokens occupy the context window on a given call — is now widely recognized as a bigger lever on output quality than prompt wording alone. Large context windows make it tempting to include everything; disciplined architecture curates instead of dumps.
| Context Component | Purpose | Design Note |
|---|---|---|
| System / role prompt | Stable behavioral contract: persona, constraints, output format | Keep static across calls to maximize prompt-cache hits |
| Retrieved context (RAG) | Grounding facts relevant to the current query | Rank by relevance; cite sources; cap token budget explicitly |
| Conversation history | Continuity across turns | Summarize or truncate older turns rather than keeping full verbatim history forever |
| Tool definitions & schemas | What actions the model can take this turn | Only include tools relevant to the current task — irrelevant tools add tokens and confusion |
| Memory (long-term) | User preferences, past decisions, durable facts | Retrieve selectively based on relevance, not injected wholesale every turn |
| Output format spec | JSON schema, XML tags, or structured grammar | Constrains generation for reliable downstream parsing |
Context Compression Techniques
Periodically compress older conversation turns into a running summary, keeping only recent turns verbatim. Trades some fidelity for a bounded context size regardless of conversation length.
Instead of keeping all history in-context, embed and store past turns, then retrieve only the turns relevant to the current query — turning conversation history into its own mini-RAG system.
Providers cache the KV-state of a repeated prompt prefix (system prompt, tool definitions, static context) so subsequent calls only pay compute for the changed suffix — a significant cost and latency lever for agentic loops that re-send similar context every turn.
Instead of injecting raw retrieved documents, pre-extract only the fields/facts relevant to the task (e.g. via a cheap extraction pass) — trading one extra small model call for a much smaller, denser context payload to the expensive call.
Caching & Cost Optimization Architecture
GenAI unit economics are dominated by tokens in and tokens out. Architecture-level caching and routing decisions typically save far more than prompt micro-optimization.
Cache the compute for a static prompt prefix (system instructions, few-shot examples, tool schemas) so repeated calls only process the new suffix. Biggest win for agent loops that resend near-identical context every turn.
Cache full responses keyed by semantic similarity of the input (not just exact match) for idempotent, non-personalized queries — FAQ-style questions are the classic case. Requires a freshness/invalidation policy tied to the underlying data.
Send the bulk of traffic to a small/cheap model; escalate only the subset that needs it to a larger model. Typically the single largest cost lever in high-volume GenAI products, often cutting cost 5-10x with minimal quality loss.
| Lever | Typical Savings | Trade-off |
|---|---|---|
| Prompt/prefix caching | Meaningful reduction on repeated-context calls | Requires stable prompt structure; cache misses on any prefix change |
| Model cascading | Often the largest single lever at scale | Requires a reliable difficulty classifier or confidence signal |
| Output length capping | Proportional to reduction in generated tokens | Risk of truncating legitimately long, necessary answers |
| Batching (async/offline workloads) | Lower per-unit cost on batch-eligible endpoints | Not usable for interactive, latency-sensitive paths |
| Retrieval-context trimming | Reduction proportional to unnecessary context removed | Requires reranking quality high enough to trust fewer, better chunks |
Multimodal Architecture Patterns
Multimodal GenAI systems combine text with images, audio, or video — either as input (a vision-language model reading a document image) or output (image/audio generation). The architecture question is where modality conversion happens: at ingestion, inside the model, or as a separate specialist step.
A single VLM/omni model directly consumes image/audio/video tokens alongside text tokens in one context window (e.g. reading a scanned invoice image and answering questions about it directly). Simplest architecture when the model's native multimodal capability covers the task.
Convert non-text modalities to text first (OCR, ASR transcription, image captioning) using specialist models, then run the rest of the pipeline (RAG, agents) purely in the text domain. Simpler to debug, cache, and index — text is easy to search and version.
For output generation (image, audio, video), route to purpose-built generative models rather than the primary reasoning LLM — an orchestrating text model calls an image-generation model as a tool, keeping reasoning and generation concerns separate.
Index images/tables/charts alongside text — via multimodal embeddings (e.g. CLIP-style) that place images and text captions in the same vector space, or by generating text descriptions of visual elements at ingestion time for standard text retrieval.
| Input Type | Common Pipeline | Key Design Note |
|---|---|---|
| Scanned documents / PDFs with layout | Layout-aware OCR or native VLM reading → structured text/JSON | Preserve table/layout structure; don't flatten tables to plain text |
| Audio (calls, meetings) | ASR transcription → diarization → text pipeline | Store timestamps and speaker labels as metadata for downstream retrieval |
| Charts and diagrams | VLM description generation → indexed alongside source image | Store both the generated description and a reference to the original image for citation |
| Video | Frame sampling + VLM captioning, or native video-understanding model | Frame sampling rate is a cost/completeness trade-off; scene-change detection beats fixed intervals |
Evaluation & Guardrails Architecture
Evaluation and guardrails are not an afterthought bolted onto a finished GenAI system — they're an architectural layer that must be designed alongside retrieval and generation, because "does this work" and "is this safe" are the two questions that determine whether a GenAI product survives contact with real users.
- Golden datasets with known-correct answers, run on every model/prompt/pipeline change
- Retrieval metrics: recall@k, MRR, NDCG for RAG pipelines
- Generation metrics: faithfulness/groundedness, answer relevance, LLM-as-judge scoring against a rubric
- Regression gate: block deploys that degrade the golden set below a threshold
- Implicit signals: thumbs up/down, regeneration rate, session abandonment
- Shadow evaluation: run a new pipeline version alongside production, compare outputs without serving them
- A/B testing for prompt, retrieval, or model changes on live traffic
- Continuous LLM-as-judge sampling on a slice of live traffic for drift detection
Guardrail Layers
| Layer | Checks | Placement |
|---|---|---|
| Input guardrails | Prompt injection detection, jailbreak patterns, PII in user input, off-topic/abuse filtering | Before the request reaches the model |
| Retrieval guardrails | Access-control filtering by tenant/role, source trust scoring, freshness checks | Between retrieval and context assembly |
| Output guardrails | Toxicity/harm classifiers, PII leakage detection, factuality/groundedness checks against retrieved sources, format validation | Before the response is returned to the user |
| Tool-call guardrails | Argument validation against schema, allowlisted operations, destructive-action confirmation | Before a tool call executes (see §17 for agent-specific guardrails) |
| Business-rule guardrails | Domain-specific hard constraints (e.g. never quote a price outside a valid range) | Deterministic post-processing, not left to model judgment |
Core Agent Design Patterns
An agent is a model wrapped in a loop that observes, decides, acts, and evaluates — repeatedly — until a goal is met or a stop condition triggers. The pattern you choose determines how the model reasons at each step and how much autonomy it has over its own next action.
The model interleaves explicit reasoning ("Thought: ...") with actions ("Action: call_tool(...)") and observations ("Observation: tool result"), repeating until it produces a final answer. The most widely used agent pattern — simple to implement, and the reasoning trace aids debuggability.
The model first produces a multi-step plan up front, then executes each step (possibly with a separate, cheaper executor model), replanning only if a step fails or new information invalidates the plan. Reduces the number of expensive planning calls versus step-by-step ReAct for long tasks.
After producing an output, the model (or a separate critic call) reviews its own work against the goal or a rubric, and revises if it falls short — looping a bounded number of times. Improves quality on tasks with a checkable success criterion at some added latency/cost per revision cycle.
Instead of committing to one reasoning path, the agent explores multiple candidate reasoning branches, scores them (via self-evaluation or a separate scorer), and expands the most promising ones — trading significant compute for better performance on tasks with many plausible-but-wrong paths.
A lightweight classifier or cheap model first determines which specialized downstream flow, prompt, or agent should handle the request, before any expensive reasoning happens. Common as the entry point of larger agentic systems (see §14).
A simple, durable loop: repeat "attempt the goal, run the verifier, feed the result back" until the verifier passes or a budget is exhausted — de-emphasizing sophisticated in-context planning in favor of a tight, deterministic-verifier-driven cycle. Popularized in coding-agent contexts (see the companion Loop Engineering Handbook for depth).
Choosing a Pattern
| Task Shape | Recommended Pattern | Why |
|---|---|---|
| Single tool call resolves it, but which tool is unclear | ReAct, single iteration | Minimal overhead; reasoning trace picks the right action |
| Long, mostly-known multi-step task (e.g. "set up a project") | Plan-and-Execute | Avoids re-planning from scratch on every step; cheaper executor for routine steps |
| Output quality matters more than latency, checkable success criterion exists | Reflection loop with a verifier | Bounded, verifier-gated revision beats a single unchecked pass |
| Many plausible solution paths, need to compare before committing | Tree of Thoughts / search | Explores alternatives instead of committing to the first plausible path |
| Long-running autonomous task with a clear pass/fail gate (tests, build) | Ralph loop / run-to-goal | Deterministic verifier is more trustworthy than in-context self-assessment |
Tool Use & Function Calling Patterns
Tool use is what turns a model from a text generator into an agent that can act on the world — query a database, call an API, run code, browse the web. The architecture of the tool layer determines reliability, safety, and how easily new capabilities can be added.
The model provider's API accepts a list of tool schemas (name, description, JSON-schema parameters) and returns structured tool-call requests instead of free text when it decides to act. The standard, most reliable mechanism — parsing is deterministic, not regex-on-free-text.
An open, standardized protocol for exposing tools/resources to models across providers and hosts — a connector written once can be reused across agent frameworks and even across model vendors, rather than writing bespoke tool-calling glue per integration.
Instead of exposing many narrow tools, give the model a sandboxed code interpreter and let it write code to accomplish tasks (data manipulation, math, chained API calls) — trading a larger, more general capability surface for more careful sandboxing requirements.
Expose the RAG retriever itself as a callable tool ("search_docs(query)") rather than always injecting retrieved context automatically — letting the agent decide when it actually needs to look something up, which is the core of Agentic RAG (§04).
Tool Design Principles
get_order_status(order_id) is far more reliable than a generic query_database(sql) tool — narrow tools constrain the model's action space, are easier to validate, and are easier to permission granularly.
Tool name, description, and parameter descriptions are read by the model exactly like prompt text — ambiguous or incomplete tool descriptions cause misuse just as ambiguous instructions do. Write them with the same care as a system prompt.
Design tools so repeated identical calls are safe (or explicitly guard against duplicate side effects) — agent loops sometimes retry or repeat calls, and non-idempotent tools (e.g. "charge_card") need explicit deduplication/confirmation logic.
Tool results returned to the model should be concise and structured (not a raw 50KB API dump) — truncate, summarize, or paginate large results before they re-enter the context window on the next loop iteration.
Agent Memory Architectures
LLMs are stateless between calls — every form of "memory" an agent has is architecture, not a model capability. Memory design determines whether an agent gets smarter over time or relearns the same lessons in every session.
The current context window — conversation history, current task state, scratchpad from the active reasoning loop. Bounded by context window size; managed via truncation, summarization, or selective retrieval (see §07).
A record of past interactions/sessions — what happened, what was decided, outcomes. Typically stored as embeddings for semantic retrieval ("has this user asked something like this before?") plus structured metadata (timestamps, outcomes).
Durable facts distilled from experience — user preferences, learned domain facts, corrected mistakes — kept separate from raw episodic logs and periodically consolidated/deduplicated so it doesn't grow unbounded.
Reusable "how to do X" knowledge — a SKILL.md-style file, a learned tool-use pattern, or a cached successful plan for a recurring task type — loaded into context when a matching task type recurs, so the agent doesn't rediscover the same procedure from scratch.
| Memory Type | Storage Pattern | Write Trigger | Read Trigger |
|---|---|---|---|
| Working | In-context, ephemeral | Every turn | Every turn (current context) |
| Episodic | Vector store / log DB, timestamped | End of session/task | Semantic similarity to current query |
| Semantic | Structured store (KV, small DB), curated | Explicit distillation step, human-reviewed ideally | Relevance to current task type |
| Procedural | Versioned skill files / prompt library | After a successful, generalizable task completion | Task-type match at loop start |
Multi-Agent Orchestration Patterns
Multi-agent architectures split work across specialized agents instead of asking one agent/prompt to do everything. The payoff is separation of concerns, parallelism, and the ability to use different models per role — the cost is coordination complexity and harder debugging.
A central orchestrator agent decomposes the task, dispatches subtasks to specialized worker agents, and integrates their results. The most common and easiest-to-debug multi-agent pattern — the orchestrator is a single point of coordination and observability.
Agents run in a fixed sequence, each consuming the previous agent's output (e.g. researcher → writer → editor). Simple, predictable, easy to reason about — but rigid; doesn't adapt if a downstream agent discovers upstream work needs revisiting.
One agent (often a faster/cheaper model) produces work; a structurally separate agent (often a stronger model, different instructions) verifies it against a spec or rubric. The agent that wrote the output should never be the sole judge of its own correctness.
Dispatch the same or related subtasks to multiple agents in parallel (e.g. independent failing tests fixed by separate agents in isolated worktrees), then merge results. Cuts wall-clock time for decomposable, independent work.
Multiple agents argue different positions or independently solve the same problem, with a judge agent (or human) selecting or synthesizing the best answer — useful for reducing single-model blind spots on ambiguous or high-stakes decisions, at multiplied cost.
Agents don't communicate directly — they read and write to a shared state store ("blackboard"), each contributing when it has relevant expertise. Flexible for loosely-coupled agents but harder to reason about causality and debug than direct orchestration.
Coordination Mechanisms
| Mechanism | How It Works | Trade-off |
|---|---|---|
| Direct orchestrator calls | Orchestrator invokes each agent as a function/subagent call, gets a return value | Simple, synchronous; orchestrator is a bottleneck and single point of failure |
| Message queue / event bus | Agents publish/subscribe to events (task created, result ready) | Decoupled, scalable, async — harder to trace a single task's full lifecycle |
| Shared document / state store | Agents read/write a common artifact (plan doc, shared memory) | Good for loosely coordinated work; race conditions possible without locking |
| Durable workflow engine | A workflow engine (e.g. Temporal-style) tracks state, retries, and step history explicitly | Strong reliability and resumability guarantees; more infra to operate |
End-to-End Agentic System Architecture
A production agentic system composes every pattern above into one coherent architecture. Below is a reference topology for a general-purpose agentic assistant — the shape most production systems converge on regardless of domain.
Orchestration & Workflow Engines
The orchestration layer is where you encode how an agent or multi-agent system moves between steps — and it's the layer most prone to becoming an unmaintainable tangle of if/else logic if not deliberately architected as a graph or state machine from the start.
Model the workflow as a directed graph of nodes (agent calls, tool calls, conditional branches) and edges (transitions, possibly conditional on state). Frameworks like LangGraph make branching, looping, and cyclical agent behavior explicit and inspectable rather than buried in imperative code.
For long-running or business-critical agentic workflows, a durable execution engine (Temporal-style) persists workflow state at every step, enabling automatic retry, resumption after a crash, and long-running (hours to days) workflows without holding state in process memory.
For non-agentic pipelines (fixed RAG Q&A, a summarize-then-format pipeline), a plain sequential chain of calls is sufficient and simpler to debug than a full graph engine — don't reach for graph orchestration when a straight-line pipeline suffices.
Steps trigger on events (new document ingested, webhook received, scheduled interval) rather than a synchronous call chain — natural fit for loop-engineering-style autonomous systems that discover their own work (see the companion Loop Engineering Handbook).
State Management Across Turns
| State Element | Where It Lives | Why It Matters |
|---|---|---|
| Conversation/task state | Explicit state object passed through the graph (not implicit in message history alone) | Makes branching logic and conditionals legible and testable |
| Tool-call results | Attached to state, not just re-derived from raw message history | Enables downstream nodes to reference structured results directly |
| Budget counters | Explicit fields (turns used, cost spent, time elapsed) | Enforceable stop conditions independent of model behavior |
| Human-in-the-loop flags | Explicit interrupt/checkpoint state | Allows pausing a graph mid-execution for approval, then resuming from that exact point |
Guardrails, Safety & Human-in-the-Loop
Agentic systems introduce safety concerns beyond single-call generation, because they can take actions, not just produce text. This section covers the agent-specific layer on top of the general guardrails in §10.
Every agent and every tool it can call should operate under least-privilege scope — read-only until proven reliable, write access limited to exactly the operations the task requires. An agent should never hold broader permissions than a junior engineer would for the equivalent task.
Insert mandatory human approval before high-stakes or irreversible actions (production deploys, financial transactions, external communications) — implemented as an explicit graph node that pauses execution and waits for approval, not a suggestion the agent can talk itself out of.
Hard caps on turns, tool calls, cost, and wall-clock time per task, plus scope limits on which files/systems/records an agent can touch — enforced outside the model's control so a reasoning failure can't cascade into an unbounded resource or damage event.
Treat all content the agent reads from untrusted sources (web pages, emails, documents, tool outputs) as potentially adversarial input that may contain embedded instructions — structurally separate system instructions from retrieved/tool content, and never let content read by a tool silently expand the agent's permissions.
Escalation Design
| Trigger | Escalation Action | Urgency |
|---|---|---|
| Budget (turns/cost/time) 50% consumed | Notify owner asynchronously (Slack, dashboard) | Low — informational |
| Budget fully exhausted without success | Stop loop, summarize partial progress, page owner | Medium |
| Same failure repeats N consecutive times | Stop loop immediately — repetition signals a structural blocker, not transient noise | High |
| Forbidden action attempted (scope violation) | Kill the agent run immediately, alert security/on-call | Critical |
| Destructive action pending confirmation | Block and wait for explicit human approval before proceeding | Always required, not urgency-based |
Observability & Evaluation for Agents
Agentic systems fail in ways traditional software doesn't — a plausible-sounding but wrong intermediate reasoning step, a tool called with subtly wrong arguments, a loop that "succeeds" against a weak verifier. Observability architecture for agents needs to capture the full reasoning trace, not just inputs and outputs.
- Every model call: prompt, response, tokens, latency, cost
- Every tool call: name, arguments, result, execution time
- Every reasoning/planning step, not just the final output
- Full state snapshots at each orchestration graph transition
- Turns/steps per completed task (trending down = the system is improving via memory/skills)
- Tool-call success/error rate, by tool
- Human intervention rate (target: trending down over time)
- Cost and latency per completed task, not just per call
- Loop termination reason distribution (success / budget / escalation)
| Observability Layer | Tooling Pattern | Primary Consumer |
|---|---|---|
| Distributed tracing | OpenTelemetry spans per model/tool call, correlated by session/task ID | Engineers debugging a specific failed run |
| Structured trace store | Purpose-built LLM/agent observability platform (trace explorer, prompt diffing) | Engineers iterating on prompts/pipelines |
| Aggregate dashboards | Cost, latency, success-rate, escalation-rate time series | Team leads monitoring system health |
| Offline eval harness | Golden dataset re-run on every pipeline/prompt/model change | CI gate before deploy |
| Online eval sampling | Continuous LLM-as-judge scoring on a slice of live traffic | Drift and regression detection in production |
Deployment & Scaling Architecture
Deploying GenAI and agentic systems combines familiar distributed-systems concerns (scaling, isolation, retries) with GenAI-specific ones (long-running non-idempotent tool calls, streaming, variable-latency model calls).
Short RAG Q&A fits a synchronous request/response model with streaming. Multi-step agentic tasks that can run minutes-to-hours need an async job model: submit task → return a job ID → client polls or subscribes for status/streaming updates → final result delivered via webhook/callback.
Each agent execution (especially code-execution or file-modifying agents) should run in an isolated sandbox/container/worktree so parallel runs don't collide and a compromised or buggy run can't affect others — mirroring the worktree-isolation pattern from loop engineering.
The orchestration/agent-execution layer should be stateless and horizontally scalable, with task state persisted externally (workflow engine, database) — not held in a single long-lived process — so orchestration workers can be added, removed, or restarted without losing in-flight tasks.
Both inbound (protect the system from being overwhelmed) and outbound (respect model-provider rate limits, avoid cascading retries that amplify load) rate limiting are needed — with queueing and backpressure signals rather than silent request drops.
Environment & Release Strategy
| Environment | Model/Prompt Policy | Guardrail Posture |
|---|---|---|
| Development | Latest experimental prompts/models, frequent iteration | Relaxed, but PII/secrets rules still enforced |
| Staging | Candidate release, full golden-set eval required before promotion | Full guardrail suite active, mirrors production |
| Production (canary) | New version served to a small traffic percentage, shadow-compared | Full guardrails + heightened monitoring/alerting |
| Production (stable) | Fully rolled out version, pinned model/prompt versions | Full guardrails, standard monitoring |
Reference Architecture & Decision Matrix
Use this section as a quick-decision reference when scoping a new GenAI or agentic system — matching problem shape to the architecture patterns covered above.
Problem-to-Pattern Decision Matrix
| If your problem is... | Reach for... | Section |
|---|---|---|
| Answers need to be grounded in a knowledge base that changes often | RAG (advanced or agentic, depending on query complexity) | §04 |
| The model needs a consistent behavior/format that prompting can't nail down | SFT / LoRA fine-tuning | §05 |
| Retrieval quality is the bottleneck, not generation | Hybrid retrieval + reranking, better chunking | §04, §06 |
| Cost is unsustainable at current volume | Model cascading + prompt caching | §08 |
| The task needs multiple sequential decisions with tool use | ReAct or Plan-and-Execute agent | §11 |
| Output quality needs an extra quality pass | Reflection loop with a verifier | §11 |
| One agent can't cover the task's breadth or needs independent review | Orchestrator-worker or implementer/verifier split | §14 |
| The system needs to run unattended for hours with checkpoints | Durable workflow engine + graph orchestration | §16 |
| The system can take real-world actions (write, send, delete) | Scoped tool permissions + HITL checkpoints + budgets | §12, §17 |
| You can't tell why a run failed | Full trace logging + agent-specific metrics | §18 |
Maturity Model for GenAI/Agentic Systems
- Single model call, naive RAG
- No eval suite
- Minimal guardrails
- Manual testing only
- Advanced RAG, hybrid retrieval
- Golden-set offline evals
- Input/output guardrails live
- Basic tracing and dashboards
- Tool-using agent loop
- Verifier-gated completion
- Budget limits and escalation
- Full trace + agent metrics
- Multi-agent orchestration
- Durable workflow execution
- Distilled memory/skills layer
- Continuous eval + auto-rollback