Senior AI QA Engineer / SDET — Hybrid Berlin

Creandum; United States; Hybrid; USD 78,000 - 117,000; Full time

Benefits offered by this job

Learning budget

Work-from-home stipend

Equity

Vacation days

Job summary

Leapsome is hiring a QA Engineer to own the quality of our LLM-powered features and help build an AI-native SDLC. You will drive eval coverage, gate features, and coach squads to write tests and evals for their own features, embedding quality into every team.

You will develop our AI QA agent and autonomous systems, define risk tiers with product/design/engineering, and ensure CI is fast, cost-efficient, and trustworthy while guarding sensitive customer data.

Qualifications

• Software engineer who specializes in quality and writes production-quality TypeScript.
• Build systems that automate and improve testing, and encode fixes into agents.
• Strong eye for UI/UX; raise flow inconsistencies early in specs.
• Experience with evals for LLM features and AI agents.
• Experience running AI agents for testing or review in a team.
• Maintain a high bar for customer data privacy and security.
• Think in terms of risk, evidence and cost while improving quality and speed.
• Have led engineering teams in elevating quality practices.

Responsibilities

• Own the quality of our LLM-powered features: eval coverage per feature, validated datasets, a gate every new AI feature passes before it merges, and clear routing to the owning squad when an eval fails
• Coach squads to write and maintain evals and tests for their own features, so quality is built into every team
• Develop our AI QA agent and other autonomous systems as a product: benchmark specs, regression protection, fast cloud runs, and repeatable release checks
• Define risk tiers with product, design and engineering to decide verification levels
• Make CI fast, cost-efficient and trustworthy: manage runner/model cost, pipeline duration, and fix flaky tests
• Refine our quality standards in the skills and rules our coding and review agents follow
• Review specs for testability before development starts, including MCP server or agent actions
• Measure quality with metrics like escaped defects, severity, and agent/evals reliability

Skills

Quality engineering

TypeScript

UI/UX

LLM evals

AI testing

Automation

Tools

Playwright

Playwright MCP/CLI

Vitest

Mocha/Chai

GitHub Actions

Job description

Leapsome is hiring a QA Engineer to own the quality of our LLM-powered features and help build an AI-native SDLC. You will drive eval coverage, gate features, and coach squads to write tests and evals for their own features, embedding quality into every team.

You will develop our AI QA agent and autonomous systems, define risk tiers with product/design/engineering, and ensure CI is fast, cost-efficient, and trustworthy while guarding sensitive customer data.