Connect
Register an agent endpoint, its available tools, policy documents, and the environment it is expected to operate in.
Independent build · Harness engineering · Agent evaluation infrastructureUnder development
AgentProof.
A stateful evaluation harness for testing AI agents before they are trusted with real customers, tools, and business data.
- Role
- Product, AI systems, full-stack engineering
- Initial domain
- Customer-support agents
- Current focus
- Go evaluation engine and stateful sandbox
- Status
- Research and development
The problem
An agent can sound correct while doing the wrong thing.
AI agents are moving beyond conversation. They retrieve private records, call APIs, issue refunds, cancel orders, and make choices that alter real systems. A fluent response does not prove that the right tool was selected, the correct record was changed, or a company policy was followed.
Teams need a repeatable way to test behaviour before release and after every prompt, model, or tool change. The evaluation must inspect both the conversation and its consequences.
Product thesis
Evaluate what the agent did, not only what it said.
AgentProof places an agent inside a controlled but realistic business environment. It simulates users, records every decision, verifies the resulting data, and makes failures reproducible.
Simulate
Run realistic and adversarial users through multi-turn scenarios backed by a sandboxed business state.
Evaluate
Grade language quality, tool selection, policy compliance, and the actual state changes produced by the agent.
Compare
Replay the same suite against new prompts or models and expose regressions before a production release.
System design
Built as an evaluation system, not a chatbot wrapper.
Scenario engine
Builds stateful users, edge cases, and adversarial conversations from controlled templates.
Tool sandbox
Provides realistic APIs for orders, refunds, cancellations, account data, and human escalation.
Execution runner
Coordinates multi-turn sessions while capturing messages, tool calls, latency, tokens, and cost.
Evaluation engine
Combines deterministic assertions, state diffs, policy rules, and calibrated model-based graders.
Trace explorer
Replays every decision so a developer can understand a failure instead of receiving only a score.
Evaluation framework
Multiple forms of evidence, one understandable result.
Task completion
Did the agent resolve the user’s actual request?
Tool correctness
Did it choose the right tool and provide valid arguments?
State integrity
Did the underlying system end in the expected state?
Policy compliance
Did the agent respect permissions, limits, and escalation rules?
Groundedness
Were its claims supported by retrieved records and policies?
Experience quality
Was the response clear, useful, and appropriate for the situation?
Safety by design
Realistic enough to reveal failure. Isolated enough to fail safely.
- Sandboxed tools and synthetic business records
- Explicit permissions and prohibited-action checks
- Deterministic verification of every state mutation
- Prompt-injection and sensitive-data scenarios
- Human review for ambiguous evaluation results
- Complete audit trail for every test run
Build roadmap
From focused test environment to reusable platform.
Support-agent sandbox
Orders, refunds, cancellations, policy retrieval, escalation, and a first set of reproducible scenarios.
Evaluation and replay
State assertions, model graders, trace inspection, regression runs, and prompt-to-prompt comparison.
Bring your own agent
External agent endpoints, MCP tools, custom policies, private scenario suites, and team reports.