AI - Software Development Engineer in Test (AI - SDET)

web; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

web is seeking a Quality Engineer to build quality practices for AI-powered applications, spanning language model workflows, copilots, retrieval systems, and autonomous agents. You’ll collaborate with engineering, AI, platform, product, and operations teams to enhance reliability, safety, and scale.

Responsibilities include designing evaluations for quality and safety, creating automated tests and benchmarks, testing agent memory and tool use, and connecting quality checks to CI/CD pipelines

Qualifications

• Strong Python programming skills and a solid foundation in software testing.
• Experience testing AI or language model systems, prompt design, evaluation methods, retrieval systems, or AI agents.
• Experience with API, UI, microservice, and data validation, including SQL.
• Familiarity with CI/CD, containers, Kubernetes, and at least one cloud platform.
• Strong analytical, debugging, and troubleshooting skills; experience working in Agile teams.

Responsibilities

• Design evaluations for response quality, factual grounding, bias, toxicity, context relevance, consistency, and safety.
• Create automated checks, test datasets, and benchmarks for language model outputs, agent workflows, and retrieval-augmented generation systems.
• Test agent memory, planning, tool use, multi-agent coordination, and integrations with external tools and protocols.
• Build and maintain Python-based UI, API, and service automation; connect quality checks to CI/CD pipelines.
• Validate retrieval components, model performance, latency, cost, throughput and behavior in distributed or cloud environments.
• Use observability data to investigate failures and improve quality practices across the development lifecycle.

Skills

Python

Quality engineering

Software testing

Agile

Debugging

Tools

SQL

CI/CD

Docker

Kubernetes

Cloud platforms

Job description

Role overview

Build quality practices for AI-powered applications, including language model workflows, copilots, retrieval systems, and autonomous agents. You’ll work across engineering, AI, platform, product, and operations teams to assess system behavior and develop automation that supports reliability, safety, and scale.

Responsibilities

• Design evaluations for response quality, factual grounding, bias, toxicity, context relevance, consistency, and safety.
• Create automated checks, test datasets, and benchmarks for language model outputs, agent workflows, and retrieval-augmented generation systems.
• Test agent memory, planning, tool use, multi-agent coordination, and integrations with external tools and protocols.
• Build and maintain Python-based UI, API, and service automation; connect quality checks to CI/CD pipelines.
• Validate retrieval components, model performance, latency, cost, throughput and behavior in distributed or cloud environments.
• Use observability data to investigate failures and improve quality practices across the development lifecycle.

Requirements

• Strong Python programming skills and a solid foundation in software testing and quality engineering.
• Practical experience testing AI or language model systems, prompt and context design, evaluation methods, retrieval systems, or AI agents.
• Experience with API, UI, microservice, and data validation, including SQL.
• Familiarity with CI/CD, containers, Kubernetes, and at least one cloud platform.
• Strong analytical, debugging, and troubleshooting skills; experience working in Agile teams.

Nice to have

• Experience with Robot Framework, Playwright, vector databases, embeddings, or agent development frameworks.
• Knowledge of model lifecycle testing, performance or security testing, AI observability, or AI red teaming.
• A portfolio of AI or automation projects, open-source contributions, or technical writing.
• A relevant degree is preferred; equivalent practical experience and demonstrable projects are also valued.

Benefits and work setup

Work may be remote, hybrid, or office-based. The listing describes learning and development opportunities, paid time off, annual bonus, health coverage, tuition and internet reimbursement, and other benefits; eligibility and details may vary.