AI QA Engineer: Eval Automation for Multimodal Agents

Hivemind

Brussel Hoofdstad

On-site

EUR 65,000 - 90,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Napster is seeking a QA Engineer on the Professional Services team to own the quality of AI agents and experiences shipped to our strategic customers. You will design evals, build UI test automation for web, mobile, and kiosk experiences, and wire tests into our deployment pipelines.

You will collaborate with Forward Deployed Engineers, Solution Engineering, Enablement, Product and Engineering, perform manual exploratory testing on hardware when needed, and travel up to 15% to customer sites.

Qualifications

  • 4+ years of QA engineering or SDET experience.
  • Experience designing evals for LLM-based products, such as golden datasets and rubric scoring.
  • Strong test automation skills in Python and/or TypeScript.
  • Experience with UI automation frameworks (Playwright, Cypress, Selenium, Appium) and CI/CD integration.
  • Daily use of agentic coding tools such as Claude Code, Cursor, or Codex.
  • Solid understanding of how LLM agents work, including tools, memory, and failure modes.

Responsibilities

  • Design, build, and maintain AI evaluation suites for Napster agents and digital twins; cover accuracy, grounding, tool usage, persona fidelity, and safety.
  • Build golden datasets and test cases with customers and FDEs; convert conversations into regression sets.
  • Implement LLM-as-judge and rubric-based scoring; calibrate against human review and track quality metrics.
  • Build and own UI test automation for web, mobile, and kiosk front ends; use Playwright/Cypress/Appium.
  • Automate end-to-end and API tests for agent integrations and real-time sessions; ensure latency checks.
  • Integrate evals into CI/CD so deployments have automated quality gates and reports.

Skills

QA engineering
SDET
Test automation
Python
TypeScript
Playwright
Cypress
Appium
CI/CD

Tools

Playwright
Cypress
Selenium
Appium
GitHub Actions
GitLab CI

Job description

Napster is seeking a QA Engineer on the Professional Services team to own the quality of AI agents and experiences shipped to our strategic customers. You will design evals, build UI test automation for web, mobile, and kiosk experiences, and wire tests into our deployment pipelines.

You will collaborate with Forward Deployed Engineers, Solution Engineering, Enablement, Product and Engineering, perform manual exploratory testing on hardware when needed, and travel up to 15% to customer sites.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Quality Assurance Engineer
Quality Assurance Engineer

Hivemind • Brussel Hoofdstad

On-site
EUR 65,000 - 90,000
AI Quality & Evaluation Engineer for Secure AI Systems
AI Quality & Evaluation Engineer for Secure AI Systems

Delen Private Bank • Antwerpen

On-site
EUR 60,000 - 80,000
Hospitalisation insurance
Dental insurance
Group insurance
+1
Senior QA Architect & Test Automation Coach
Senior QA Architect & Test Automation Coach

Select Group • Deinze

On-site
EUR 65,000 - 90,000
Competitive salary
Benefits package
Team events and social activities
Hybrid QA Lead: AI-Driven Cloud Testing
Hybrid QA Lead: AI-Driven Cloud Testing

Dstny • Brussel Hoofdstad

Hybrid
EUR 70,000 - 100,000
Health insurance
Life insurance
Meal allowance €10.46/day
+4
QA Automation Coach
QA Automation Coach

Select Human Resources NV • Oost-Vlaanderen

On-site
EUR 60,000 - 85,000
Dynamic and international work enviro​
Competitive salary and attractive bene
QA & Testing Presales Lead — Quality Engineering Expert
QA & Testing Presales Lead — Quality Engineering Expert

NTT America, Inc. • Brussel Hoofdstad

On-site
EUR 60,000 - 85,000
BeFlex policy
Career development & mentorship
Global Top Employer recognition
+1
Test Engineer
Test Engineer

Nova Hunte Ltd • Brussel Hoofdstad

On-site
EUR 50,000 - 70,000
Software Engineer - AI-Driven Test Automation Engine
Software Engineer - AI-Driven Test Automation Engine

Anite • Brussel Hoofdstad

On-site
EUR 63,000 - 104,000
Global QA Lead: Test Automation & CI/CD
Global QA Lead: Test Automation & CI/CD

Select Human Resources NV • Oost-Vlaanderen

On-site
EUR 60,000 - 85,000
Dynamic and international work enviro​
Competitive salary and attractive bene
AI Agent & Automation Engineer – Copilot Studio & Python
AI Agent & Automation Engineer – Copilot Studio & Python

Editx • Brussel Hoofdstad

On-site
EUR 70,000 - 100,000