How to Test AI Agents in Production: Tool Calls, Evals, Failure Injection & MCP Security

An agent saying “Your tour is booked” is not enough evidence. The test must establish that the right calendar contains one booking for the right customer, that the caller was authorized, and that the final reply agrees with the result. Start in an isolated test environment, then use production observations to improve the suite.

Define an outcome before writing a grader

For a scheduling workflow, define allowed calendars, required confirmation, timezone behavior and the meaning of a completed booking. Separate a confirmed success, a confirmed failure and an unknown outcome. An HTTP timeout can leave the effect unknown if the provider committed the write before the connection failed.

Anthropic’s guide to agent evaluations distinguishes tasks, repeated trials, graders and execution traces. A practical consequence is to evaluate both the trajectory and the resulting environment state. Deterministic checks are useful for observable effects; judgment-based graders need calibration against human review.

Build a small fixture set around expensive failures

• Happy path: one authorized booking and a response consistent with its receipt.
• Wrong parameters: another tenant’s calendar or an unavailable slot is rejected.
• Provider rejection: no booking and no claim of success.
• Timeout after commit: reconcile the existing operation before retrying.
• Duplicate delivery: the same operation does not create a second booking.
• Injected instructions: retrieved content cannot authorize exporting customer data.
• Partial completion: a booking succeeds but its notification fails; recovery must not book again.

A runnable assertion example

Save this dependency-free example as agent-outcome.test.mjs and run node --test agent-outcome.test.mjs. It demonstrates a grader for normalized execution evidence. Replace the sample trace with observations captured by your integration adapter; do not ask the model to report its own calls or effects.

import assert from "node:assert/strict";
import test from "node:test";

function assertBookingOutcome(trace) {
  assert.equal(trace.calls.length, 1);
  assert.equal(trace.calls[0].tool, "create_booking");
  assert.equal(trace.calls[0].calendarId, "tenant-a-calendar");
  assert.equal(trace.receipt.status, "confirmed");
  assert.equal(trace.bookings.length, 1);
  assert.equal(trace.bookings[0].id, trace.receipt.bookingId);
  assert.equal(trace.bookings[0].calendarId, "tenant-a-calendar");
  assert.equal(trace.bookings[0].contactId, "contact-a");
  assert.equal(trace.replyOutcome, "confirmed");
}

const success = {
  calls: [{ tool: "create_booking", calendarId: "tenant-a-calendar" }],
  receipt: { status: "confirmed", bookingId: "booking-1" },
  bookings: [{ id: "booking-1", calendarId: "tenant-a-calendar",
    contactId: "contact-a" }],
  replyOutcome: "confirmed",
};

test("accepts an observed booking", () => assertBookingOutcome(success));
test("rejects false success after an API failure", () => {
  assert.throws(() => assertBookingOutcome({
    ...success, receipt: { status: "failed" }, bookings: [],
  }));
});
test("rejects duplicate effects", () => {
  assert.throws(() => assertBookingOutcome({
    ...success, bookings: [...success.bookings, ...success.bookings],
  }));
});
test("rejects a cross-tenant tool call", () => {
  assert.throws(() => assertBookingOutcome({
    ...success, calls: [{ tool: "create_booking", calendarId: "tenant-b-calendar" }],
  }));
});

This example deliberately covers one narrow contract. It does not test a live agent, prove authorization enforcement or interpret free-form language. Your adapter must derive replyOutcome from the actual user-visible response, and the grader needs assertions for time, approval and operation identity appropriate to your workflow.

Inject failures at the integration boundary

Mock both an explicit API rejection and a lost response after a successful write. They require different recovery behavior. Reset fixture state between trials, keep the same operation identity across retries of one action and verify the effect through a separate read. Test concurrency against a sandbox or integration database; an in-memory mock does not prove durable deduplication.

Repeat runs and gate releases by risk

Run each critical scenario multiple times and report the number of successful trials out of the total. Keep the model version, prompt, tool definitions and fixture version with the result. A high average score must not hide a single unauthorized write. Set release gates per failure class and investigate regressions instead of silently rerunning until a test passes.

Connect observability to the next regression case

Trace conversation IDs, operation IDs, tool names, sanitized parameters, result categories and final effects. Measure task completion separately from tool execution, plus latency, cost, escalation and unknown outcomes. Restrict access and redact customer data before export. Turn reproducible incidents into synthetic fixtures and review sensitive failures with a human.

For MCP-connected agents, add the MCP security checklist to the same regression suite. Tool access and authorization need coverage alongside response quality.

Apply the approach to a real workflow

VenueX AI is the production context behind my focus on booking, messaging, guardrails and human review. The test above uses invented fixtures and makes no claim about private product test results. If your agent is approaching launch, the AI Agent Testing service turns your critical workflows into a scoped review and regression plan.