Building Agentic AI Systems

How do you turn flexible generative AI into dependable agents that act on your behalf? This book argues that you succeed by combining strong generative backbones (VAEs, GANs, Transformers) with explicit notions of agency, well-chosen architectures, grounded tool use, pragmatic planning, rich memory, reflective feedback loops, and robust trust-and-safety layers. The authors contend that powerful agents emerge when you separate language reasoning from action execution, scaffold planning to fit natural-language ambiguity, and instrument the system with monitoring and governance. But to do so, you must understand the trade-offs among model families, autonomy levels, orchestration patterns, and safety constraints.

In this guide, you discover how VAEs, GANs, and especially Transformer-based LLMs support generation across text and multimodal tasks. You then learn what makes a system “agentic”—the differences between agency, autonomy, and self-governance—and how to architect single- and multi-agent systems that balance responsiveness and reasoning. Finally, you learn to ground LLM plans with tools and controllers, structure memory and roles for consistency, add reflection loops for continual improvement, and operationalize trust, safety, and ethics in production.

Why generative backbones matter

Generative models learn probability distributions and can synthesize new content. VAEs provide structured latent spaces for controllable sampling; GANs push photorealistic outputs (e.g., StyleGAN2/3 for faces); and Transformers power natural language and multimodal generation (GPT-3/4, PaLM, DALL·E, Stable Diffusion, Flamingo). You pick the backbone that fits your problem: VAEs when you need disentangled latent factors, GANs when you prioritize visual fidelity, and LLMs when language understanding, planning, and tool reasoning dominate. In agentic systems, LLMs often sit at the core because they reason over goals and instructions, then call tools to act.

Defining agentic behavior

Agency is the capacity to act intentionally on goals; autonomy is the degree of independence; self-governance is the ability to regulate behavior without external control. The book anchors these definitions with a TripPlanner example: the agent asks clarifying questions, queries flight APIs, evaluates options, and requests permission before charging a card. You learn to calibrate autonomy—operational (execute tasks), functional (choose strategies), and hierarchical (where it sits in a larger system)—so actions remain useful and safe.

Architectures and multi-agent coordination

Architectural choice steers behavior. Deliberative agents offer explainable plans but may be slow; reactive agents deliver real-time robustness but lack long-horizon reasoning; hybrid agents blend both. For complex work, you often assemble multi-agent systems (MASs) that coordinate, cooperate, and negotiate. The Coordinator-Worker-Delegator (CWD) model maps human organizational roles to AI agents, helping you scale via separation of concerns and resilient delegation (e.g., a travel Coordinator sets strategy, a Delegator assigns tasks, Workers execute flight/hotel/activity searches).

Planning that fits language

Classical planners (STRIPS, A*, GraphPlan, MCTS) excel with explicit states but struggle with natural language fuzziness and the cost of LLM rollouts. The most practical pattern is hierarchical planning—HTN-like decomposition (plan, then execute)—guided by the LLM. You can adapt FF heuristics by having the model sketch relaxed plans, then refine them. The core rule: use structure to constrain search, and LLMs to interpret ambiguous goals. Plan-then-execute workflows keep reasoning distinct from tool calls and simplify error handling.

Grounding with tools and controllers

Without tools, LLMs hallucinate. The book shows how to define tools with clear docstrings or JSON schemas and route calls through an Agent Controller that validates parameters, handles auth, manages rate limits, retries on failure, and logs actions. Frameworks map this differently—CrewAI (@tool decorators, manager–worker flow), AutoGen (round-robin chats, termination rules), LangGraph (graph-based orchestration with memory). The pattern is the same: the LLM proposes; the controller executes safely.

Memory, roles, and reflection

Consistent behavior requires layered memory—short-term (working), long-term (profiles and knowledge), and episodic (interaction history)—plus crisp role prompts and backstories that shape preferences and constraints. Reflection and meta-reasoning close the loop: agents self-explain, monitor their strategies, adjust internal weights, and update self-models over time. You get adaptive teammates, not just chatty tools.

Trust, safety, and governance

Autonomous action raises stakes. The authors emphasize explainability (attention maps, saliency, natural-language rationales), uncertainty display, debiasing, privacy protections (differential privacy, federated learning), and rollback paths. Action boundaries and human-in-the-loop checkpoints ensure interventions in high-risk contexts. Taken together, these measures turn agentic systems into reliable, auditable products that regulators and users can trust.

Key Idea

Agentic systems succeed when you pair language-native planning and reflection with grounded tool execution, structured memory, disciplined roles, and strong governance—separating thinking from doing while keeping both transparent and controllable.

The book starts with model families that make generation possible and shows you when each one fits. VAEs, GANs, and Transformers all learn data distributions but in different ways. If you match their strengths to your goals, you reduce complexity and cost while improving results. Crucially, agentic systems often lean on Transformers for reasoning and planning, but VAEs and GANs still shine in structured control and realism (especially in vision and design pipelines).

VAEs: structured latent spaces

A VAE learns an encoder–decoder pair that maps data to a continuous latent space and back, trained to reconstruct inputs while regularizing the latent distribution. Because the latent space is smooth and probabilistic, you can sample or interpolate to generate coherent outputs. Variants like Beta-VAE emphasize disentanglement (useful for controllable features), and CVAE conditions on attributes to guide generation. The book cites drug discovery work (e.g., AstraZeneca) where VAEs enable molecular design with property control—an early example of agency-adjacent generation.

GANs: adversarial realism

GANs pit a generator against a discriminator in a minimax game. DCGAN leverages convolutional inductive biases; WGAN stabilizes training with the Wasserstein distance and gradient penalties; StyleGAN2/3 deliver unmatched facial realism and style control. The trade-off is training instability and evaluation complexity—great for high-fidelity images, tricky to optimize. In agentic pipelines, GANs often serve as a rendering or enhancement step after a planner or VAE proposes structure (e.g., concept-to-image flows).

Transformers: language and multimodality

Transformers use self-attention to capture long-range dependencies and scale to billions of parameters. Autoregressive models (GPT-3/4, PaLM) predict the next token, enabling fluent text and plan generation; encoder-only models (BERT) excel at understanding; encoder–decoder (T5) supports text-to-text reframing. Modern multimodal systems (DALL·E, Stable Diffusion, Flamingo) extend this backbone to images and audio. For agentic systems, Transformers enable tool reasoning, chain-of-thought, and hierarchical plan decomposition—bridging from fuzzy user goals to executable actions.

Choosing the right backbone

If you require interpretable control, VAEs and CVAEs give you knobs in latent space; if photorealism matters, GANs still lead but demand careful engineering; if reasoning, dialog, and heterogeneous tools dominate, go with LLMs. Compute and data cost also guide choices—GANs can require large, curated image datasets; LLMs demand diverse corpora and strong alignment (instruction tuning, RLHF). In all cases, align your backbone to the downstream agent architecture and tool stack to avoid brittle bridges.

From generation to action

Generation alone is not agency. The book shows how you wrap generation with intent, planning, and controlled execution. For example, a travel agent LLM doesn’t just “write an itinerary”; it parses goals, checks dates and budgets, calls a flight search tool, evaluates options, and requests approval before booking. Here the Transformer’s strength is orchestrating reasoning and language-based planning, while external tools ground facts and commit changes (the Agent Controller is the gatekeeper).

Key Idea

Pick VAEs for structured control, GANs for visual fidelity, and Transformers for language-native reasoning and multimodal orchestration—then fuse them with tools to convert generation into reliable actions.

(Note: While diffusion models dominate modern image generation, the book’s focus on GAN families remains useful for understanding adversarial training dynamics and style control. You can treat diffusion backbones as complementary in the same agentic scaffolds.)

You build better systems when you name what “agentic” means and bound it. The book clarifies three pillars: agency (capacity to act intentionally on goals), autonomy (degree of independence), and self-governance (ability to regulate and update behavior). With this vocabulary, you decide when an assistant may act, how it should adapt, and what approval gates it must honor. These decisions materially affect safety, user trust, and liability.

Agency: acting for users

An agent acts on behalf of a principal, holding decisional authority within constraints. The TripPlanner in Chapter 1 exemplifies agency: it elicits constraints (dates, budget), queries flight/hotel APIs, scores options, and can initiate bookings. The model’s intentionality shows up in goal tracking and option evaluation—not just generating text. Importantly, the agent remains responsible for outcomes within its autonomy band, which shapes auditing and rollback design.

Autonomy: levels and levers

Operational autonomy lets agents execute well-scoped actions; functional autonomy gives freedom to select strategies; hierarchical autonomy defines its place in organizational control. You might grant operational autonomy for itinerary searches, functional autonomy for choosing scoring criteria, but keep hierarchical autonomy low by requiring user confirmation to charge payments. These levers let you right-size independence to domain risk and compliance needs (think healthcare vs. travel).

Self-governance: adapting within bounds

Self-governance includes self-organization, regulation, adaptation, and optimization. Concretely, the TripPlanner maintains goals, updates a knowledge base, and uses an internal rule—select the option with max(score)—to decide. You can extend this with learned preference weights and policy gradients in safe sandboxes. The key is to make updating explicit and observable so you can explain behavior and revert regressions.

Designing constrained autonomy

Even when you enable autonomous booking, you add guardrails: permission prompts for payments, timeouts, action limits per session, and domain-specific verifiers. The agent may retry failed tools with backoff but must escalate after N attempts. In practice, autonomy is tiered: low-stakes tasks (date parsing) run fully automated, mid-stakes tasks (rebooking) require soft confirmation, and high-stakes tasks (financial commitments) enforce hard approval.

Key Idea

Grant just enough autonomy to deliver value and never more than you can supervise—tie every action privilege to explicit oversight, logging, and rollback paths.

(Note: This framing echoes safety patterns from human–automation interaction literature and aligns with “constrained delegation” in modern agent frameworks.)

Architecture determines how your agent thinks and responds. The book walks through deliberative, reactive, and hybrid styles, then scales up to multi-agent systems (MASs) with explicit interaction mechanisms. A practical organizational metaphor—the Coordinator-Worker-Delegator (CWD) model—helps you separate concerns, scale throughput, and preserve reliability under load. You learn to match styles and roles to problem demands instead of forcing a one-size architecture.

Deliberative, reactive, and hybrid

Deliberative agents follow sense–plan–act loops with rich knowledge representations and predictable decisions; they suit constraint-heavy or safety-critical tasks but can be slow. Reactive agents map perceptions to actions directly and shine in noisy, real-time settings (e.g., sensors, micro-decisions) but lack long-horizon planning. Hybrid systems combine a reactive layer for quick responses with a deliberative layer for strategy—ideal for dialog agents that must be responsive while orchestrating multi-step tool calls.

Scaling with multi-agent systems

MASs decompose work across specialized agents that cooperate (shared goals), coordinate (manage interdependencies), or negotiate (auction-style resource allocation). In travel, airline, hotel, and ground transportation agents collaborate to assemble packages. The book outlines request handling, package compilation, and confirmation loops you can port to LangGraph or AutoGen. Attention to heterogeneity, distributed control, and robust communication protocols prevents deadlocks and improves fault tolerance.

CWD: organizational clarity

The CWD model assigns strategy to Coordinators, task distribution to Delegators, and execution to Workers. In the travel example, a Coordinator (Planning Executive) decomposes goals; a Delegator (Operations Orchestrator) assigns tasks to Worker specialists (Aviation Booking, Hospitality Expert, Destination Curator, Ground Logistics); results flow up for integration and user presentation. Delegators also enforce tool access and action boundaries, and can reassign tasks on failure to maintain SLA targets.

Framework mappings

CrewAI naturally supports manager–worker patterns with @tool decorators per role. AutoGen offers group chats (RoundRobinGroupChat) with explicit termination conditions, great for collaborative refinement. LangGraph models the workflow as a state graph with agent and tool nodes plus MemorySaver checkpoints, which is ideal for persistent, resumable runs. Choose by orchestration needs: if you want explicit dataflow and checkpoints, pick LangGraph; if you want hierarchical role prompts, pick CrewAI; for ad hoc multi-agent dialog, pick AutoGen.

Design Principle

Map architecture to task demands: deliberative for complex constraints, reactive for low-latency robustness, hybrid for most agentic dialogs; then scale with MAS and CWD to separate strategy, orchestration, and execution.

(Note: This echoes classic robotics hybrids like subsumption + planning, applied to LLM-era software. You gain both speed and explainability.)

Agents need more than pattern-matching—they need knowledge structures and reasoning methods. This section explains semantic networks, frames, and logic-based representations, then pairs them with deduction, induction, and abduction. You also see how learning modes—supervised, unsupervised, reinforcement, and transfer—fit into agent lifecycles. In practice, you combine these ingredients so agents can understand, explain, and adapt reliably.

Semantic networks and frames

Semantic networks represent concepts as nodes and relations as typed edges (Dog → is-a → Animal). Frames act like structured objects with slots and default values (Vehicle/Car), supporting inheritance and procedural attachments for computed attributes. With these, your agent infers properties (e.g., a Car inherits Vehicle max_speed) and triggers code when slots change—crucial for consistent decision logic.

Logic for guarantees

Propositional, first-order, and temporal logic provide formal semantics and inference guarantees. You might encode rules like ∀x(Human(x) → Mortal(x)) to ensure sound conclusions. Logic shines in legal or safety-critical domains where verifiability matters. The cost is engineering complexity and brittleness in open-text environments, which is why many systems blend light-weight logic with LLM flexibility (a pragmatic middle ground).

Reasoning styles you actually use

Deduction confirms consequences from rules (great for verification and constraint checking). Induction learns generalizations from data (ML models). Abduction proposes best explanations for observations (diagnostics). You often cycle them: abduct a hypothesis, test deductively, refine inductively. In the travel agent, abductive steps hypothesize why a route is best (price vs. layover), then deductively verify constraints (dates, loyalty benefits) before committing.

Learning mechanisms in practice

Supervised learning maps inputs to labels; unsupervised discovers latent structure; reinforcement learning (RL) learns policies via rewards; transfer learning fine-tunes models for new domains. RL works best when you can simulate interactions (games, robotics), while transfer learning speeds domain adaptation for LLMs (instruction tuning for travel dialogs). Generative AI multiplies your options with synthetic data and language-based knowledge encoding.

Why this matters

Good knowledge representation plus the right reasoning style yields predictable, explainable agent behavior—while learning mechanisms and synthetic data keep it improving.

(Note: Compared to purely end-to-end LLM systems, this approach reintroduces modularity, making it easier to verify and debug decisions—similar to neuro-symbolic patterns in other research.)

Tools turn words into actions. The book treats tool use as a first-class design problem: define clear capabilities, communicate schemas to LLMs, and route every action through an Agent Controller that handles validation, auth, and error recovery. With this separation, the LLM proposes well-typed calls (e.g., search_flights, bookFlight), and the runtime executes safely—closing the loop from intent to effect without trusting the model to act directly.

Defining tools the model understands

CrewAI’s @tool decorator uses docstrings to convey what a tool does, required inputs, and outputs. You can also expose JSON schemas when calling models directly. The trick is to treat docs as instructions: be explicit about parameters (location, date), constraints (rate limits), and examples of calls. Clear schemas reduce misuse and enable validation before hitting external APIs.

The Agent Controller contract

An Agent Controller mediates between LLM suggestions and the world. It enforces security, parameter checks, retries, circuit breakers, and logging. When a tool fails, it returns structured errors for the LLM to handle or for a Delegator to reassign. This design keeps reasoning (LLM) and execution (runtime) separate, improving testability and compliance. In the TripPlanner, the controller also manages tokens, sessions, and PII redaction to prevent leaks.

Framework patterns you can reuse

CrewAI ties tools to role prompts and supports manager–worker orchestration. AutoGen coordinates multi-agent chats, letting specialists call tools and hand off work with termination rules. LangGraph explicitly models agent/tool nodes and state transitions with MemorySaver checkpoints—ideal for long-running workflows and resilience. Regardless of framework, prioritize idempotent tool design, structured returns, and observability dashboards.

Grounding to avoid hallucinations

Without tools, LLMs produce plausible fabrications. The book’s CrewAI travel planner illustrates this: it writes a convincing but imaginary itinerary when disconnected. Grounding via search APIs, booking platforms, and databases converts narratives into verifiable actions. Combine tool reasoning prompts (“consider which tool is necessary next”) with schemas and examples to raise selection accuracy and sequence correctness.

Practical Insight

Treat tools as products: document them, version them, validate their inputs, and monitor their outcomes—your agent is only as reliable as its toolchain and controller.

(Note: This mirrors serverless patterns—functions as clean, testable boundaries—now orchestrated by an LLM planner instead of hard-coded flow logic.)

Planning is where agents turn goals into sequences of actions. The book contrasts classical planners with LLM-based approaches and recommends hybrids that handle natural-language uncertainty. You learn why HTN-style decomposition and plan-then-execute loops map well to LLMs, how to adapt FF heuristics, and when to reach for classical guarantees. The outcome is a playbook for picking the right planner for your domain.

Classical planning: strong, but brittle in language

STRIPS, A*, GraphPlan, and FF rely on explicit predicates, operators, and search. They yield provable plans and are superb in discrete, well-modeled domains (robot motion, logistics). But language introduces fuzzy states and unbounded actions, making classical representations expensive to craft and maintain. MCTS suffers too because each rollout would require multiple LLM calls—computationally heavy.

LLM-based planning: flexible decomposition

LLMs parse vague goals (“relaxing beach vacation with culture”) and propose stepwise decompositions. In CrewAI examples, a Travel Planning Strategist drafts subgoals, and a Researcher validates with tools. You still need guardrails: ground steps with APIs, enforce schemas, and verify constraints before committing. The language-native fit is powerful—you get plans that feel human and adapt to ambiguity.

HTN: the practical middle

Hierarchical Task Networks decompose tasks into subtasks until they reach primitive actions. That mirrors chain-of-thought prompting and manager–worker delegation. You can encode reusable task hierarchies (flight → hotel → activities) while letting LLMs fill gaps (supplier-specific quirks). Depth control and explicit subgoal schemas keep the search bounded and auditable.

FF adaptations: relaxed plans via LLM

The book proposes an LLMFastForward sketch: ask the LLM for a relaxed plan—ignoring certain delete effects—to quickly approximate a path, then choose the next action to execute. This keeps search affordable while leveraging LLM fluency. It is “moderately practical” because you still need engineered mappings from natural language to executable operators.

Plan-then-execute with tool grounding

A robust pattern is: instruct the LLM about available tools (with schemas), request an explicit plan referencing those tools, and let the Agent Controller execute steps with validation and retries. If a step fails, either repair the plan locally or ask the LLM to replan. This separation keeps reasoning clean and execution safe across frameworks like LangGraph (graph nodes), CrewAI (manager–worker), and AutoGen (conversation-driven).

Practical Rule

Use classical planners for discrete, fully modeled domains; prefer HTN-structured, LLM-guided planning for language-rich, uncertain tasks—always grounded by tools and controllers.

(Note: This echoes broader industry shifts toward “programs that write programs,” with LLMs drafting plans that traditional runtimes execute deterministically.)

Agents become reliable partners when they remember, role-play consistently, and learn from experience. The book details memory types, system prompts with roles and backstories, and reflection/meta-reasoning loops that adjust strategies over time. Together, these components turn a one-off conversation into an evolving collaboration—especially valuable in domains like travel, where preferences accumulate and exceptions recur.

Layered memory for continuity

Short-term (working) memory holds the immediate session context (current query, active searches). Long-term memory stores persistent profiles and knowledge (CustomerMemory, TravelKnowledge: loyalty status, seat preferences, seasonal norms). Episodic memory records interaction histories to resurface successful patterns and avoid past mistakes. An event-driven state model updates bookings on flight changes or weather alerts and validates dependencies (flight → hotel → transfers) before committing.

Roles, backstories, and templates

System prompts define an agent’s role, goals, constraints, and communication style. Backstories encode domain heuristics (e.g., a former airline revenue manager prioritizes schedule reliability and loyalty accrual). Standardized input/output schemas reduce parsing friction when agents pass work around (task_type, parameters; status, result). In practice, these narratives and templates act like soft rules that keep teams coherent across sessions.

Reflection and meta-reasoning

Reflection makes reasoning visible and improvable. Agents self-explain (“I chose Hotel du Petit Moulin because price fits budget and location is central”) and monitor their strategies, adjusting preference weights based on feedback (e.g., bumping “adventure” from 0.33 to 0.34). Self-modeling maintains goals and knowledge updates for persistence. Over time, this yields tailored recommendations and faster convergence on user intent.

Operationalizing adaptation

Implement reflective loops with clear triggers (user ratings, cancellations, NPS changes), safe learning rates, and audit logs. Persist updates in long-term stores and use episodic retrieval to recall similar cases. Couple self-explanations with uncertainty estimates to ask for clarification when confidence dips. This prevents drift and ensures changes remain attributable and reversible.

Key Idea

Memory keeps context, prompts enforce roles, and reflection improves policy—combine all three to transform an LLM into a dependable, evolving teammate.

(Note: This is conceptually similar to retrieval-augmented generation plus policy refinement, adapted here for agent workflows and multi-agent coordination.)

Agentic systems earn trust when you make their reasoning visible, quantify uncertainty, mitigate bias, and protect privacy and IP. Because agents can act autonomously, their mistakes carry operational and ethical risks. The book offers a toolbox—explanations, monitoring, boundaries, and rollback—to ship systems that are both useful and acceptable to users and regulators.

Explainability you can show

Use attention visualizations (e.g., BERT heads), saliency maps, and natural-language rationales (e.g., GPT-4o-mini) to explain why an option was chosen. Pair this with confidence scores and alternatives so users can compare trade-offs. In travel, you might highlight which query phrases drove the hotel pick and present two backup itineraries with uncertainty bands on price volatility.

Uncertainty and bias

Always admit uncertainty: ask clarifying questions for vague prompts, provide ranges instead of single numbers, and defer to humans when confidence is low. Combat bias with diverse data, adversarial tests, human oversight, and continuous monitoring. Because agents act, bias can become automated harm (e.g., unfair pricing). Build detectors and post-hoc audits to catch patterns early.

Action boundaries and rollback

Define what the agent may never do, where it must seek approval, and how to revert actions. Enforce decision verification for high-cost steps (payments), maintain idempotent external calls, and keep detailed logs for forensics. Integrate RLHF-style feedback loops to reinforce safe, helpful behavior without dulling capability.

Privacy, security, and IP

Use differential privacy, federated learning, and secure multiparty computation to minimize PII exposure. Segment secrets, rotate keys, and sanitize tool outputs. For IP, track provenance, watermark generated content (e.g., SynthID analogs), and audit datasets to avoid accidental infringement. In multi-agent chains, prevent cross-session data leakage with strict state isolation.

Safety Measures

Action boundaries, verification gates, uncertainty display, bias monitoring, privacy-by-design, and robust rollback are non-negotiable when agents can act autonomously.

(Note: These practices align with emerging AI governance frameworks and will likely become table stakes for regulated deployments.)