Lead QA Engineer - AI Systems for Industrial Autonomy

Nexxa.AI; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Nexxa.AI is building the best AI systems for heavy industries and seeking a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act autonomously in industrial settings.

You'll design evaluation frameworks, build datasets, and drive metrics beyond accuracy, ensuring safe, reliable agent behavior across planning, tool use, and execution. This role collaborates with ML and backend teams and requires strong Python scripting and testing expertise.

Qualifications

• 5+ years in QA/SDET roles, with ownership of test strategy for complex systems.
• Hands-on experience testing LLM-based products, chatbots, or AI agents.
• Experience with eval frameworks or tooling (promptfoo, DeepEval, RAGAS, LangSmith) or building your own.
• Strong scripting in Python to build test automation, data pipelines, and eval tooling.
• Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
• Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
• Familiarity with LLM-specific failure modes: hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.
• Comfortable operating in ambiguity — defining what "correct" means for a task when there's no single right answer.
• Strong written communication skills for turning fuzzy quality signals into clear, actionable findings for engineering and product stakeholders.

Responsibilities

• Design evaluation harnesses and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
• Develop golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial and operational contexts.
• Define and track quality metrics beyond simple accuracy — groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations.
• Build automated pipelines that run evals on every model, prompt, or tool-integration change, and integrate them into CI/CD.
• Conduct structured red-teaming and adversarial testing (prompt injection, jailbreaks, tool misuse, unsafe actions) in partnership with security teams.
• Test agent behavior across the full action loop — planning, tool selection, tool execution, error recovery, and final output — not just the final response.
• Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic.
• Partner with ML and backend engineers to translate eval failures into actionable, reproducible bug reports.
• Establish quality bars and sign-off criteria for new agent capabilities before they reach customer environments.
• Mentor other engineers on testing strategies specific to probabilistic, LLM-driven systems.
• Advocate for testability and observability in agent architecture from day one.

Skills

QA ownership

LLM testing

Python scripting

Eval tooling

LLM architecture

Test data design

Failure modes

Ambiguity handling

Technical writing

Tools

promptfoo

DeepEval

RAGAS

LangSmith

Job description

Nexxa.AI is building the best AI systems for heavy industries and seeking a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act autonomously in industrial settings.

You'll design evaluation frameworks, build datasets, and drive metrics beyond accuracy, ensuring safe, reliable agent behavior across planning, tool use, and execution. This role collaborates with ML and backend teams and requires strong Python scripting and testing expertise.