Lead QA Engineer - AI Agent Systems

Engg; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Nexxa.AI in San Francisco is seeking a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act autonomously in industrial environments. This role goes beyond UI testing and focuses on evaluating non-deterministic, tool-using systems with robust data and metrics.

You will design evaluation frameworks, build golden datasets, and drive automations that measure reasoning, tool use, and safety.

Qualifications

• 5+ years in QA/SDET with complex systems.
• Hands-on experience testing LLM-based products.
• Experience with eval tooling or building your own.
• Strong Python for test automation.
• Understanding of how LLM agents work: prompting, tool calls, context management, RAG, memory, and orchestration.
• Comfortable operating in ambiguity — defining what 'correct' means for a task with no single right answer.
• Strong written communication to engineers and product stakeholders.

Responsibilities

• Design evaluation harnesses and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
• Develop golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial contexts.
• Define and track quality metrics beyond simple accuracy — groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations.
• Build automated pipelines that run evals on every model, prompt, or tool-integration change, and integrate them into CI/CD.
• Conduct structured red-teaming and adversarial testing with security teams.
• Test agent behavior across the full action loop — planning, tool selection, tool execution, error recovery, and final output.
• Investigate and triage failures where root cause could be the model, the prompt, the tool/API, or the orchestration logic.
• Partner with ML and backend engineers to translate eval failures into actionable, reproducible bug reports.
• Establish quality bars and sign-off criteria for new agent capabilities before reaching customers.
• Mentor other engineers on testing strategies for probabilistic, LLM-driven systems.
• Advocate for testability and observability in agent architecture from day one.

Skills

QA ownership

LLM testing

Python scripting

Eval frameworks

SDET experience

Tools

promptfoo

DeepEval

RAGAS

LangSmith

Langfuse

Job description

Nexxa.AI in San Francisco is seeking a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act autonomously in industrial environments. This role goes beyond UI testing and focuses on evaluating non-deterministic, tool-using systems with robust data and metrics.

You will design evaluation frameworks, build golden datasets, and drive automations that measure reasoning, tool use, and safety.