Lead QA Engineer - Autonomous AI Agents for Industrial Ops

Nexxa.ai; Sunnyvale (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

Nexxa.AI in Sunnyvale seeks a Lead / Senior / Staff QA Engineer to own quality for ai agent systems that plan, call tools, and act autonomously in industrial environments. This role focuses on evaluation infrastructure beyond traditional UI testing.

You will design evaluation frameworks for non-deterministic systems, build golden datasets, and drive regressions while collaborating with ML, backend, and Forward Deployed Engineers to define what good looks like and measure it continuously.

Qualifications

• 5+ years in QA/SDET roles with ownership of test strategy.
• Hands-on experience testing LLM-based products or AI agents.
• Experience with eval frameworks or building your own.
• Strong Python scripting for automation and data pipelines.
• Understanding prompting, tool calling, context management in LLMs.

Responsibilities

• Design evaluation harnesses and regression suites for LLM-based agents.
• Develop golden datasets and labeled test sets including edge cases.
• Define quality metrics beyond accuracy and monitor safety.
• Build CI/CD pipelines for evals on model/tool changes.
• Conduct red-teaming and adversarial testing with security teams.
• Test agent behavior across planning, tool use, and recovery.
• Triager failures and translate findings into bug reports.
• Establish quality bars and sign-off criteria for new capabilities.
• Mentor engineers on probabilistic QA strategies.
• Advocate for testability and observability from day one.

Skills

QA ownership

LLM testing

Eval frameworks

Python scripting

Prompting & tool calls

Dataset design

Non-determinism handling

Communication

Tools

promptfoo

DeepEval

RAGAS

LangSmith

Langfuse

Job description

Nexxa.AI in Sunnyvale seeks a Lead / Senior / Staff QA Engineer to own quality for ai agent systems that plan, call tools, and act autonomously in industrial environments. This role focuses on evaluation infrastructure beyond traditional UI testing.

You will design evaluation frameworks for non-deterministic systems, build golden datasets, and drive regressions while collaborating with ML, backend, and Forward Deployed Engineers to define what good looks like and measure it continuously.