Start with one narrow support task, connect the Anthropic API to approved knowledge, and require human review before the agent can take consequential action. Log every request, tool call, source and outcome so you can test the system against real tickets before expanding its authority.
In April 1970, Apollo 13’s crew faced rising carbon dioxide after an oxygen tank failure forced them into the lunar module. The command module carried square lithium hydroxide canisters; the lunar module used round ones. NASA engineers in Houston had to develop a way to connect incompatible equipment using materials available aboard the spacecraft, then communicate the procedure to the crew.
NASA’s Apollo 13 history documents the improvised adapter and the uncertainty surrounding the mission. The rescue depended on a constrained solution: a known problem, an inventory of available materials, a procedure that could be tested on the ground, and people retaining control of the decision.
That mechanism maps closely to a first customer support agent. Reliability comes from narrowing the problem, controlling the available tools, testing the procedure and defining when a human takes over.
Choose one support job with a clear boundary
Do not begin with “automate customer support.” Pick a task whose inputs and acceptable outputs you can describe precisely.
A useful first job might classify incoming tickets, retrieve relevant documentation and draft a reply for an agent to approve. Password resets, refunds, account suspensions and billing changes carry more risk because an incorrect action can affect access or money. Keep those outside the first release unless the agent only gathers information and routes the case.
Write a short operating contract before touching the API:
- What information may the agent read?
- What may it write or change?
- Which requests must go to a person?
- What evidence must support an answer?
- What should happen when sources conflict or provide no answer?
This contract becomes the basis for prompts, permissions and tests. It also exposes vague requirements early. “Answer accurately” cannot be tested until you define the approved sources, required citations and escalation conditions.
Build a controlled path from ticket to response
Create the smallest end-to-end integration. Send a sanitized ticket to the Anthropic API with a system instruction that defines the support role, allowed sources, response format and refusal conditions. Ask for structured output containing the ticket category, proposed answer, supporting source, confidence or uncertainty note, and escalation reason.
Connect only the tools required for the chosen job. Anthropic has announced production-agent tooling that includes computer use, the Skills API and the Files API. Treat each capability as a separate permission decision.
The Files API can support access to approved documents, but file availability does not establish that a document is current or authoritative. Maintain an explicit knowledge set with owners and revision dates. The Skills API can package repeatable procedures, but each procedure still needs tests and version control. Computer use introduces broader operational risk because the agent can interact with interfaces designed for people. Keep it out of the first support workflow unless the task genuinely requires it, and restrict the reachable systems and actions when you introduce it.
Put an application layer between the model and production systems. That layer should validate inputs, remove unnecessary personal data, enforce tool permissions, cap retries and record trace identifiers. Never place API credentials in prompts, ticket text or client-side code.
If a tool can issue credits, cancel service or change account access, require an approval checkpoint. The lesson from an unauthorised agent payment applies directly: a plausible model response is not authorization.
Test failures before measuring speed
Build an evaluation set from representative, redacted support cases. Include ordinary questions, incomplete requests, conflicting documentation, angry messages, prompt injection attempts and cases that require escalation.
Score outcomes against observable criteria:
- Did the agent choose the correct category?
- Did it use an approved source?
- Did the answer match that source?
- Did it avoid unsupported claims?
- Did it escalate restricted or ambiguous cases?
- Did it refrain from calling tools it did not need?
Run the set whenever you change the model, system prompt, knowledge files, skill definitions or tool permissions. Model output can vary, so one successful demonstration proves little. Review repeated runs and inspect the failure distribution.
Log the model version, prompt version, retrieved material, tool arguments, tool results, final response, latency and human decision. Redact sensitive fields before storing them. These records let you distinguish a retrieval failure from a reasoning error, a permission problem or outdated support documentation.
Release through a human queue
Deploy first in shadow mode, where the agent processes live tickets without affecting customers. Compare its proposed routing and replies with what support staff actually did.
Next, let it draft responses for human approval. Track acceptance, edits, escalation accuracy and harmful near misses. A high acceptance rate can still conceal a serious authorization flaw, so review the rejected cases individually.
Only grant autonomous actions after the corresponding task has passed a defined test set and has a reversible failure path. Set limits per tool, preserve an audit trail and provide a kill switch. For production changes suggested by an agent, use the same evidence checks described in verifying an agent’s late-night fix.
Apollo 13’s improvised adapter worked because Houston tested a bounded procedure against the materials the crew actually had. Give your support agent the equivalent: a narrow job, an explicit inventory of permitted tools, rehearsed failure cases and a person ready to take control. Start with one queue and one approval screen, then review the first batch of disagreements before adding another capability.
Comments
No comments yet.