AI Quality Engineer: Evaluation Pipelines & LLM Testing

Dealstitch LLC.; United States; On-site; USD 160,000 - 220,000; Full time

Job summary

Dealstitch LLC is seeking an AI Quality Engineer to own the evaluation system for production-grade agents. You will run and extend pipelines that validate agents in client engagements and our internal fleet, ensuring honesty as models and data evolve.

You will measure success with automated gates in CI/CD, score agents at step-level, and expand the toolkit for failures, safety, and observability across production deployments.

Qualifications

• Strong Python programming and testing for non-deterministic systems.
• Experience evaluating LLM/agent systems: agent graphs, eval frameworks, and LLM-as-a-judge methods.
• Fluency with retrieval quality (RAG), observability and tracing for production agents.
• A bias for measurement: define what "good" means and prove it with a harness.
• Bonus: data-pipeline testing or exposure to private markets or enterprise data.

Responsibilities

• Own the evaluation pipelines that gate every change we ship, automated in CI/CD.
• Score agents at the step level across tool selection, planning, reasoning chains, and retrieval quality.
• Extend the eval toolkit for new failure modes: LLM-as-a-judge, RAG faithfulness, hallucination detection, and red-teaming for safety.
• Maintain golden datasets and tracing/observability for live agents.

Skills

Python

Testing non-deterministic systems

CI/CD

LLM eval frameworks

Observability/Tracing

Job description

Dealstitch LLC is seeking an AI Quality Engineer to own the evaluation system for production-grade agents. You will run and extend pipelines that validate agents in client engagements and our internal fleet, ensuring honesty as models and data evolve.

You will measure success with automated gates in CI/CD, score agents at step-level, and expand the toolkit for failures, safety, and observability across production deployments.