Lead QA Engineer - Autonomous AI Agents for Industry

Nexxa.AI; Sunnyvale (CA); On-site; USD 140,000 - 200,000; Full time

Benefits offered by this job

Competitive compensation

Equity package

Job summary

Nexxa.AI in Sunnyvale, CA seeks a Lead/Senior/Staff QA Engineer to own quality for our AI agent systems that plan, call tools, and take multi-step actions autonomously in industrial environments.

You will design evaluation infrastructure, build golden datasets, and measure non-deterministic reasoning quality, safety, and latency. This role partners with ML/backend teams to raise the bar for trustworthy autonomous systems.

Qualifications

• 5+ years in QA/SDET roles, with demonstrated ownership of test strategy for complex systems.

Responsibilities

• Design and build evaluation harnesses and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
• Develop golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial and operational contexts.
• Define and track quality metrics beyond simple accuracy — groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations.
• Build automated pipelines that run evals on every model, prompt, or tool-integration change, and integrate them into CI/CD.
• Conduct structured red-teaming and adversarial testing (prompt injection, jailbreaks, tool misuse, unsafe actions) in partnership with security teams.
• Test agent behavior across the full action loop — planning, tool selection, tool execution, error recovery, and final output — not just the final response.
• Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic.
• Partner with ML and backend engineers to translate eval failures into actionable, reproducible bug reports.
• Establish quality bars and sign-off criteria for new agent capabilities before they reach customer environments.
• Mentor other engineers on testing strategies specific to probabilistic, LLM-driven systems.
• Advocate for testability and observability in agent architecture from day one.

Skills

QA strategy

LLM testing

Python scripting

Eval frameworks

Adversarial testing

CI/CD automation

Documentation

Education

Bachelor's degree in CS or related

Tools

promptfoo

LangSmith

RAGAS

Langfuse

Weights & Biases

Job description

Nexxa.AI in Sunnyvale, CA seeks a Lead/Senior/Staff QA Engineer to own quality for our AI agent systems that plan, call tools, and take multi-step actions autonomously in industrial environments.

You will design evaluation infrastructure, build golden datasets, and measure non-deterministic reasoning quality, safety, and latency. This role partners with ML/backend teams to raise the bar for trustworthy autonomous systems.