Project

Agent Evaluation Benchmark

A confidence-aware evaluation framework for openclaw-class personal assistants, built to measure more than demo performance.

Python · Wilson CI · Cohen's kappa · Bootstrap · Browser Automation

2026-present

Product Snapshot

Role: Independent research and evaluation engineering
Focus: A four-pillar confidence model for browser-use and personal-assistant agents, plus broader real-workflow coverage.
Validation: Applied statistical confidence, judge agreement, coverage, and cross-run stability; a separate expansion added 40 RPA, IM, cross-system, and email tasks.
Public Proof: 120 labeled tasks and 17 Chrome scripts; a 50-task batch logged 1,152 actions and 1,020 screenshots, plus 72 run audits. The 40 expansion tasks are integrated but not fully run-validated.

Next Step

Portfolio PDF About the Founder Email

Why I started it

As agents move from chat to task execution, the evaluation problem changes completely.

I wanted a benchmark that could answer a harder question than “it looks capable”: how reliable is this agent, and under what uncertainty?

The framework

The structure I designed uses four pillars:

statistical confidence under limited samples
judge agreement instead of assuming the evaluator is always correct
coverage tied to real task distributions
stability across repeated runs

What matters most

My favorite part of the project is the claim that coverage should not aim for uniformity by default. It should resemble the real distribution of important tasks, otherwise a benchmark can look fair while hiding the behaviors that matter most in production.

Evidence in use

The framework has moved beyond a proposal. Current assets include 120 labeled tasks and 17 Chrome automation scripts. One 50-task batch logged 1,152 action steps and 1,020 screenshots, while 72 run-level audit records support cross-run stability checks.

The benchmark is still expanding, but its goal is not a polished leaderboard. It is an evaluation method that exposes confidence intervals, judge disagreement, coverage gaps, and run-to-run variation.

Real-workflow expansion

In a separate LexBench-Browser workstream, I used a cognitive-complexity matrix and probe tasks to test whether scenarios were actually executable, then designed and integrated:

10 RPA tasks
10 instant-messaging tasks
10 cross-system tasks
10 email tasks

These 40 cases are present in the browseruse-bench dataset, but the evidence supports design and integration only, not completed benchmark results. I also do not claim the baseline set of 210 tasks as my individual work.