Why a Custom AI Agent Is More Than One Prompt: KB, Tools, and Handoff
Many quotes reduce “custom AI Agent” to “write a system prompt + pick a model.” Two weeks after launch, three failures show up together: prices and policies changed but answers did not; users asking for order/CRM actions get invented stories; escalations have no handoff and the frontline screenshots the chat for a manager. Prompts own tone and role—not fact freshness, write permissions, or safety nets. GeonAI delivers on knowledge, a tool gateway, and handoff rules as the minimum surface; /agents are capability references only (not a public trial of 363+ presets). Email [email protected]. Boundary tables: /blog/custom-agent-scope-boundary-in-contract. Artifacts: /blog/custom-agent-delivery-artifact-checklist.
What prompts can and cannot do
| Capability | Prompt is roughly enough | Needs another mechanism |
|---|---|---|
| Tone and role | Yes | — |
| Current prices / policies | No (goes stale) | RAG + versions |
| Order lookup / CRM writes | No | Tool gateway + allowlist |
| High-risk promises (price, refund, lead time) | No (unreliable) | Human confirm / handoff |
| ACL and classification | No | ACL + refuse policy |
Treating “can chat” as “customized” bets operations risk on a paragraph. Buy and accept mechanisms you can update—not prompt word count.
Leg 1 — Knowledge (facts)
- Policies, SKU notes, FAQs, and price lists live in an updatable store—not hard-coded in the prompt
- Answers require citations (doc version / section); missing evidence → refuse or hand off
- Name who ingests, who approves, and how to roll back—bind versions to the release pack
Enterprise RAG: /blog/enterprise-rag-knowledge-base-agent. RAG vs fine-tuning: /blog/rag-vs-fine-tuning-enterprise. Fine-tuning helps format and tone; it does not reliably replace changing facts.
Leg 2 — Tools (actions)
- Function Call / MCP through an allowlisted gateway; ban universal “any API / SQL” tools
- Read calls need receipt codes; writes default to human confirm or a ticket
- Joint-test logs, error codes, and idempotency ship as artifacts—not a verbal “we connected it”
Tool calling and ERP/CRM patterns: /blog/agent-function-call-mcp-integration. Without a tool layer, the Agent is a talkative FAQ that cannot get work done.
Leg 3 — Handoff (safety)
- Triggers: emotion escalation, missing evidence, Table-C actions, repeated failures, insufficient ACL
- Fixed copy + ticket fields (intent, docs retrieved, tools called, user ask)
- “Fully automatic” with no handoff is unfit for customer-facing promises or write paths
Handoff is not failure—it is part of the spec. Track missed-handoff and wrong-promise rates separately; see /blog/poc-to-production-agent-checklist.
Minimum chain that ties the three legs
- Classify the intent (do / don’t / human-confirm)
- Retrieve knowledge; check citations and ACL
- (Optional) Call allowlisted tools; validate receipts
- Produce a cited answer or draft
- If rules hit, hand off; otherwise show or send on the allowed channel
When read/write duties conflict, split a read-only Agent from a draft Agent—see /blog/multi-agent-vs-single-assistant.
Buyer acceptance questions
- Who updates the KB, and where is the version id?
- Where is the tool allowlist and confirm policy?
- Where are handoff rules and ticket fields?
- Does the eval set include refuses, ACL leaks, and missing-evidence cases?
- Are the three boundary tables in the contract annex?
If the vendor cannot answer, the “custom” offer is usually a wrapped prompt—demo-friendly, audit-hostile.
How to brief GeonAI
List intents to automate, intents to refuse, systems to connect, and the three mis-ops you fear most. Email [email protected], /pricing, or Live chat. We deliver the three legs—not “one prompt and done.”
Frequently asked questions
Can an internal toy run on prompts only?
For personal productivity or a closed pilot, maybe. For customer-facing channels, writes, or price/promise paths—no.
Can fine-tuning replace the knowledge base?
Not for changing facts. Fine-tuning helps format and tone; facts need RAG with versions.
If we have a KB, do we still need tools?
Yes. The KB answers “what is true”; tools execute “look up / change.” Retrieval alone cannot complete order or CRM actions.
Won’t handoff hurt auto-resolve rate?
It hurts inflated resolve rates and cuts incident rates. Contracts should exclude Table-B refuses and compliant handoffs from penalty metrics.
How do the three legs relate to the boundary tables?
Tables define do / don’t / confirm; legs are the runtime mechanisms. Tables live in the contract; legs live in the system.
Does a demo that only edits the prompt count as custom delivery?
No. Done means a knowledge release process, tool allowlist, handoff rules, and eval coverage.