Remote AI QA Engineer for LLMs & Autonomous Agents

web; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

web is seeking a Quality Engineer to build quality practices for AI-powered applications, spanning language model workflows, copilots, retrieval systems, and autonomous agents. You’ll collaborate with engineering, AI, platform, product, and operations teams to enhance reliability, safety, and scale.

Responsibilities include designing evaluations for quality and safety, creating automated tests and benchmarks, testing agent memory and tool use, and connecting quality checks to CI/CD pipelines

Qualifications

• Strong Python programming skills and a solid foundation in software testing.
• Experience testing AI or language model systems, prompt design, evaluation methods, retrieval systems, or AI agents.
• Experience with API, UI, microservice, and data validation, including SQL.
• Familiarity with CI/CD, containers, Kubernetes, and at least one cloud platform.
• Strong analytical, debugging, and troubleshooting skills; experience working in Agile teams.

Responsibilities

• Design evaluations for response quality, factual grounding, bias, toxicity, context relevance, consistency, and safety.
• Create automated checks, test datasets, and benchmarks for language model outputs, agent workflows, and retrieval-augmented generation systems.
• Test agent memory, planning, tool use, multi-agent coordination, and integrations with external tools and protocols.
• Build and maintain Python-based UI, API, and service automation; connect quality checks to CI/CD pipelines.
• Validate retrieval components, model performance, latency, cost, throughput and behavior in distributed or cloud environments.
• Use observability data to investigate failures and improve quality practices across the development lifecycle.

Skills

Python

Quality engineering

Software testing

Agile

Debugging

Tools

SQL

CI/CD

Docker

Kubernetes

Cloud platforms

Job description

web is seeking a Quality Engineer to build quality practices for AI-powered applications, spanning language model workflows, copilots, retrieval systems, and autonomous agents. You’ll collaborate with engineering, AI, platform, product, and operations teams to enhance reliability, safety, and scale.

Responsibilities include designing evaluations for quality and safety, creating automated tests and benchmarks, testing agent memory and tool use, and connecting quality checks to CI/CD pipelines