Senior Software Development Engineer in Test (SDET), QA - AI Systems

Nexxa.ai; Sunnyvale (CA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Equity

Job summary

Nexxa.ai is seeking a Senior SDET/QA engineer in Sunnyvale, CA to build and maintain automated evaluation frameworks for AI agents. You will develop test suites, simulators, and CI/CD quality gates to ensure robust agent performance and safety.

The role requires strong Python skills, experience with distributed systems, APIs, and LLM-based products, and the ability to work across teams on test infrastructure, evaluation methods, and failure analysis.

Qualifications

• Requires at least five years of SDET, QA, or software engineering experience focused on testing.
• Strong programming skills (Python preferred) and experience building test infrastructure.
• Experience testing distributed systems, APIs, and LLM-based products.
• Proficiency in manual and automated testing, CI/CD and containerized environments.
• Experience with evaluation methods, dataset management, and LLM agent failure modes.

Responsibilities

• Build and maintain automated evaluation frameworks, test suites, simulators, and CI/CD quality gates for AI agents.
• Define quality metrics and release criteria for meaningful deployments.
• Manage evaluation datasets and investigate failures to improve agent reliability and safety.
• Collaborate with engineering and deployment teams to enhance testing capabilities across platforms.
• Participate in hands-on exploratory, regression, and adversarial testing efforts.

Skills

Python

Test Automation

Exploratory Testing

LLM Evaluation

AI Agent Testing

CI/CD

API Testing

Integration Testing

End-to-End Testing

Test Framework Development

Dataset Management

Red Teaming

Performance Testing

Tools

Docker

Kubernetes

Observability and Tracing

Job description

Build and maintain automated evaluation frameworks, test suites, simulators, and CI/CD quality gates for AI agents, while also conducting hands-on exploratory, regression, and adversarial testing. Define quality metrics and release criteria, manage evaluation datasets, investigate failures, and collaborate with engineering and deployment teams to improve agent reliability and safety.

Requirements

Requires at least five years of SDET, QA, or software engineering experience focused on testing, strong programming skills (Python preferred), and experience building test infrastructure and testing distributed systems, APIs, and LLM-based products. Candidates should be skilled in both manual and automated testing, CI/CD and containerized environments, evaluation methods, dataset management, and LLM agent failure modes.

Key Skills

• Python
• Test Automation
• Exploratory Testing
• LLM Evaluation
• AI Agent Testing
• CI/CD
• API Testing
• Integration Testing
• End-to-End Testing
• Test Framework Development
• Dataset Management
• Red Teaming
• Performance Testing
• Docker
• Kubernetes
• Observability and Tracing

Benefits

• Equity