An AI agent is a software system that can choose among actions, observe their results, and continue until a defined completion condition is met. The useful definition is operational, not theatrical. A single model response is a model call. Several predetermined calls are a chain. A fixed sequence with branching rules is a workflow. A system that selects tools at runtime and uses their results to choose the next step is an agent. Multiple agents are a coordination pattern, not an automatic upgrade.
Learning objectives
You will build a small local agent that investigates a ticket with two deterministic tools. You will separate model suggestions from application authority, represent state explicitly, log an execution trace, enforce an allow-list, and stop after a bounded number of steps. The example is in content/ship-to-production/ai-agents.examples.
You will also learn when not to use an agent. If a workflow can be expressed as a tested function or a decision table, that is often cheaper, faster, and easier to audit. Agentic behavior is justified by a real need for runtime choice, not by the label.
Prerequisites
You should be comfortable with Python dictionaries, functions, JSON-like data, and the tool-validation ideas from Tool-Using AI. Read Beyond Basic RAG if your agent will investigate a knowledge base. No cloud account or model download is required.
Why the loop matters
The loop has five responsibilities. The model proposes what might happen next. The application checks whether that action exists, whether the arguments have the right shape, and whether the current user is allowed to request it. A tool performs a narrow operation. The result enters state as untrusted evidence. A stopping policy decides whether another step is appropriate.
This separation is the central safety boundary. A tool description is not permission. A JSON schema describes structure, not business authorization. A model can request a permitted operation with an unsafe argument, so validation needs both shape checks and policy checks. The application should own secrets, credentials, rate limits, and irreversible actions.
The word agent can obscure an important engineering choice. Ask what is variable. If the order of operations is fixed, write a workflow. If one step has a few known branches, write a state machine. If a model must select one of several read-only tools based on the evidence it just received, an agent loop may be appropriate. The smallest useful abstraction is usually the most dependable one.
Mental model: proposal versus authority
Think of the model as a fallible planner sitting outside the authority boundary. It can propose an action in a structured envelope. The orchestrator parses that envelope, validates its shape, checks policy, and decides whether to call a tool. Tool output is evidence for the next decision, not an instruction that can rewrite policy. The orchestrator owns the loop and can refuse to continue.
This mental model also clarifies state. State is not the model’s hidden memory. It is an application-owned record of the request, observations, actions, errors, budget, and stop condition. A trace is an event-oriented view of that state. Keeping the two explicit makes replay, debugging, evaluation, and deletion possible.
Toy problem / implementation
The toy agent receives an incident identifier. Its deterministic planner first calls ticket_lookup, then calls recent_events only when the ticket is marked degraded. It returns a diagnosis after the second result. The planner stands in for a language model so that the execution mechanics remain reproducible.
The state is a small record containing the request, observations, pending action, completed actions, and trace. Each tool receives validated arguments and returns plain data. The dispatcher rejects unknown tools before execution. A maximum step count prevents an accidental loop from running forever.
The implementation deliberately has no eval, shell execution, network access, or hidden global state. That makes the trust boundary visible. In a production adapter, a model client would produce a structured action proposal, but the same dispatcher and policy checks would remain in the application.
Run it yourself
From the repository root, run:
bash content/ship-to-production/ai-agents.examples/verify.shThe script uses only the Python standard library. It runs unit tests and prints the action trace. The expected trace includes ticket_lookup, recent_events, and a final stop decision. Tests assert that unknown tools and excessive steps are rejected. The result is intentionally small enough to inspect line by line.
Read the source in this order: the State dataclass, the two tool functions, the dispatch boundary, and then run. That order follows ownership. State records facts, tools do narrow work, dispatch enforces policy, and the loop decides what happens next. Reproducible examples are valuable because they let you change one boundary at a time and observe the consequence.
Inspecting the execution trace
An agent trace is more useful than a final paragraph because it lets an operator answer what happened. Record the request ID, agent version, selected tool, validated arguments, result metadata, duration, and stop reason. Do not record secrets or unrestricted document contents merely because tracing is enabled.
The first visual shows the loop as a sequence rather than an abstract brain. Observe means reading the request and available evidence. Decide means selecting from a declared action set. Validate means checking schema and authorization. Execute means calling a tool with a timeout. Record means adding a bounded event to state. Stop means returning, asking for approval, or failing safely.
The loop can revisit observation after a tool result. That is where an agent differs from a simple chain: the next action is chosen from current evidence. The distinction is useful only if the choice is real and the allowed action set is visible. If the planner always selects the same next function, use a chain and remove unnecessary model calls.
The state machine makes retry behavior explicit. A transient tool failure can return to validation with a retry budget. A policy failure should go to stop, not retry until the policy changes. A missing fact can produce a clarification request. These are different states and should not be collapsed into a generic “try again.”
A state machine also gives you testable invariants. The agent must never execute a tool that is not in the registry. It must never exceed its step budget. It must not treat a failed validation as successful evidence. It must preserve a stop reason. These invariants are more useful than a vague claim that the agent is autonomous.
Real-world application: incident triage
An incident-triage agent might read a ticket, inspect a service health endpoint, search a runbook, and draft a summary. It should not silently restart production or change access controls. A useful first version has read-only tools, a small action budget, and a human approval boundary for anything consequential.
The architecture is: API request to an orchestrator; orchestrator to model adapter; model adapter to structured action proposal; policy layer to tool registry; tool registry to isolated read-only services; results back to state; trace exporter and response formatter. Retrieval can provide runbook passages, but retrieved text is data, not an instruction channel with authority.
The trace visual represents the evidence an operator should see. A production UI might link each result to a source, show latency, and distinguish model text from tool output. It should also show when the agent stopped because it reached a boundary rather than because it solved the problem.
Start with read-only investigation. Add a tool only when you can state its input contract, authorization rule, timeout, error taxonomy, and audit event. For example, a service-health tool may accept a service identifier from an allow-list and return status plus a timestamp. It should not accept an arbitrary URL merely because a model can produce one. A runbook search should return document IDs and snippets with provenance. A ticket lookup should enforce tenant boundaries before returning content.
Human approval is a state, not a sentence in a prompt. When a proposed action is consequential, persist the proposal and evidence, show the operator the exact arguments, obtain an approval tied to that proposal, and expire it after a short period. If the arguments change, require a new approval. This avoids a common error where a person approves “restart the service” but the actual execution receives a different target.
Choosing the right abstraction
A model call is appropriate for classification, extraction, or drafting when the surrounding program controls the next step. A chain is useful when each output feeds a known next prompt. A deterministic workflow is best when business rules are stable and explainable. A tool-using agent earns its complexity when evidence determines which operation comes next. A multi-agent system should be reserved for a measurable coordination need, because it multiplies state, permissions, latency, and evaluation surface.
Measure the choice. Compare task completion, invalid action rate, latency, cost, and operator interventions. If an agent does not improve the task that motivated it, simplify the system. More steps are not evidence of deeper reasoning. A shorter trace that reaches a correct, cited result within policy is usually better than a long trace that merely looks active.
Failure modes and debugging
The most common failure is an unbounded loop. Add a maximum step count, per-tool timeout, total deadline, and a clear stop reason. A second failure is tool confusion: the model chooses a plausible tool with the wrong arguments. Use strict schemas, typed adapters, and negative tests.
Another failure is state pollution. If a tool result is appended without provenance or timestamp, the agent may treat stale evidence as current. Store source, retrieval time, and confidence where those fields are meaningful. Keep user input, retrieved content, and system policy in separate fields so later prompts can apply different trust rules.
Retries require classification. Retry a short-lived transport error when the operation is idempotent. Do not retry a denied operation, invalid argument, or irreversible mutation automatically. Add jitter and a total budget for real services. A final response should say when evidence was incomplete.
Watch for prompt injection in tool results and retrieved documents. A returned log line can contain text that looks like an instruction. The policy layer should not be changed by that text. Treat external content as data, quote it with provenance, and require the planner to select from the same allow-list after every observation. This rule matters even when a document came from an internal system.
Partial failure is normal. One tool can time out while earlier observations remain valid. The agent should mark the missing observation and either stop with uncertainty or request a human. Do not fill the gap with invented content. A good final response distinguishes “the monitor reported degraded” from “the cause is confirmed.”
Security, privacy, cost, and operations
Use least privilege for every tool. Put credentials in the service boundary, never in the prompt. Treat tool output and retrieved documents as potentially adversarial. Redact personal data from traces, define retention, and provide deletion paths. Rate-limit both users and tools. Estimate model and tool cost per run, then cap the budget.
For high-impact decisions, the agent should recommend and explain, while a person approves. Log policy decisions and version the tool registry. Monitoring should distinguish model latency, tool latency, validation failures, and stop reasons. These measurements tell you whether a problem belongs in prompts, retrieval, infrastructure, or workflow design.
Version prompts, model adapters, tool contracts, policies, and output schemas together. A trace that cannot identify those versions is hard to reproduce. Keep a replay fixture with sensitive values removed. Run it in CI against deterministic tools. For model-dependent behavior, use a golden dataset and thresholds rather than expecting byte-for-byte identical text.
Strengths
Bounded agents can handle variable investigation paths while keeping tools narrow. Explicit state makes intermediate evidence inspectable. A provider-neutral dispatcher lets a team change model vendors without rewriting the authority boundary. Deterministic tools also make evaluation and incident replay practical.
Limitations and when not to use this approach
The toy planner does not demonstrate open-ended reasoning, and a successful trace is not proof of correctness. Agents add latency, state complexity, and new failure modes. Use a normal function for a stable transformation, a chain for a stable sequence, and a workflow for known branching. Use an agent only when runtime choice is valuable enough to justify the controls.
Small local tests cannot estimate production task success, security risk, or model quality. Tool coverage may be incomplete. A read-only design is safer than a mutation-capable design but cannot prove the approval flow for writes. Treat this article as an architecture exercise and measure the real application with representative cases before launch.
Exercises
Add a read-only runbook-search tool and require a citation in the final diagnosis. Add a transient failure that succeeds on the second attempt and assert that the retry budget is respected. Add an approval state for a hypothetical restart request without executing the restart. Finally, write a golden trace and compare future versions against it.
Create a case where the ticket is healthy and verify that the agent stops after one lookup. Create a malformed action with an extra argument and ensure it is rejected. Add tenant IDs to state and prove that a ticket from one tenant cannot be read by another. Then calculate the maximum number of model calls and tool calls allowed by your budget.
Final notes
The important implementation is the boundary around the model. An agent becomes trustworthy through explicit actions, constrained state, observable transitions, and a stop policy. More autonomy is not the objective; useful work completed inside an understood operating envelope is.
Next in this path
Continue to Agent Memory to decide what state should persist, what should expire, and what should be deleted.
Sources
The research dossier is available at research/dossiers/ai-agents.md. It records the papers, specifications, and official documentation used for this article. Code tested on 2026-08-05 with Python 3.11 and the standard library.