Autonomous Agent Quality Assurance Lead

Aslan; Washington (District of Columbia); On-site; USD 120,000 - 180,000; Full time

Job summary

Aslan in Washington DC is seeking a role focused on evaluation and quality for autonomous agents. You will own daily platform operation, build regression datasets, and drive the release process for models and harnesses.

You bring hands-on experience with evals for shipped LLM or agent products, and you can defend proxy metrics when the real signal isn't available, ensuring robust evaluation in production.

Qualifications

• Strong Python
• An evals framework you've run in production: Inspect, Promptfoo, Braintrust, Arize, LangSmith or equivalent
• Familiarity with agent harness internals: memory, context assembly, tool selection
• Preference data, process supervision, or reward modeling experience
• Enough statistics to tell a regression from noise on small samples

Responsibilities

• Run the platform yourself every day and log what's wrong in enough detail that an engineer can reproduce it.
• Sample agent plans, rate them, and turn ratings into a versioned dataset.
• Diagnose whether a bad decision came from the model or from the harness handing it the wrong context, and route each to the right owner.
• Write and run checks over each agent's full history, and over properties that only hold across the whole set of running agents.
• Build a simulation harness so a week of agent operation runs overnight against a candidate build.
• Define what counts as a regression between builds. Measure both whether agents last and whether they get anywhere.

Skills

Python

Evals framework in production

Agent harness internals

Preference data / supervision / reward

Statistics for regression

Job description

Aslan in Washington DC is seeking a role focused on evaluation and quality for autonomous agents. You will own daily platform operation, build regression datasets, and drive the release process for models and harnesses.

You bring hands-on experience with evals for shipped LLM or agent products, and you can defend proxy metrics when the real signal isn't available, ensuring robust evaluation in production.