AI Quality Engineer

Dealstitch LLC.; United States; Hybrid; USD 160.000 - 220.000; Vollzeit

Zusammenfassung

Dealstitch LLC is seeking an AI Quality Engineer to own the evaluation system for production-grade agents. You will run and extend pipelines that validate agents in client engagements and our internal fleet, ensuring honesty as models and data evolve.

You will measure success with automated gates in CI/CD, score agents at step-level, and expand the toolkit for failures, safety, and observability across production deployments.

Qualifikationen

• Strong Python programming and testing for non-deterministic systems.
• Experience evaluating LLM/agent systems: agent graphs, eval frameworks, and LLM-as-a-judge methods.
• Fluency with retrieval quality (RAG), observability and tracing for production agents.
• A bias for measurement: define what "good" means and prove it with a harness.
• Bonus: data-pipeline testing or exposure to private markets or enterprise data.

Aufgaben

• Own the evaluation pipelines that gate every change we ship, automated in CI/CD.
• Score agents at the step level across tool selection, planning, reasoning chains, and retrieval quality.
• Extend the eval toolkit for new failure modes: LLM-as-a-judge, RAG faithfulness, hallucination detection, and red-teaming for safety.
• Maintain golden datasets and tracing/observability for live agents.

Kenntnisse

Python

Testing non-deterministic systems

CI/CD

LLM eval frameworks

Observability/Tracing

Jobbeschreibung

# AI Quality EngineerOverviewApplication## About the roleAn agent that demos well is not the same as an agent that works in production, so every agentwe ship is backed by evals. We're hiring an AI Quality Engineer to own that system. You'll runand extend the evaluation pipelines that prove our agents work, both in client engagements andacross our own internal agent fleet, and keep them honest as the agents, models, and dataunderneath them change.## What you'll do* Own the evaluation pipelines that gate every change we ship, keeping them automated and running in CI/CD rather than leaning on manual spot-checks.* Keep agents scored at the step level, not just the final answer, across tool selection, planning and reasoning chains, and retrieval quality, so subtle regressions surface before clients do.* Extend the eval toolkit as new failure modes appear: LLM-as-a-judge, RAG faithfulness and relevance, hallucination detection, prompt and agent regression suites, and red-teaming for safety.* Maintain the golden datasets that mirror real production inputs and the tracing and observability that watch our agents once they are live.* Own the quality bar for what ships and raise it over time with the rest of the engineering team.## What we're looking for* Strong software engineering with Python, and a feel for testing non-deterministic systems, where classic assertion-based tests are not enough.* Experience evaluating LLM and agent systems: agent graphs (e.g. LangGraph), eval frameworks (DeepEval, RAGAS, G-Eval, LangSmith, or similar), and LLM-as-a-judge methods.* Fluency with RAG and retrieval quality, plus observability and tracing for agents in production.* A bias for measurement: you define what "good" means, then build the harness that proves it.* Bonus: data-pipeline testing experience, or exposure to private markets, financial services, or enterprise data.## Why DealstitchWe're a senior team with 40+ years of combined private-market technology experience.You'll work on problems that matter, with direct access to decision-makers, and see theimpact of your work in weeks, not quarters.Apply for this position## Position details### EmploymentFull-time### WorkplaceRemote### LocationRemote (US)### DepartmentEngineering### Compensation$160k-$220k