Quality Assurance Engineer

Napster

United States

Hybrid

USD 120,000 - 170,000

Full time

12 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Napster, a frontier AI company delivering embodied and agentic AI experiences, seeks a QA Engineer on the Professional Services team. You will design AI evals, build UI and API test automation for web, mobile, and kiosk experiences, and ensure production readiness with automated quality gates.

You will collaborate with Forward Deployed Engineers, Solution Engineering, Enablement, Product, and Engineering to test multilingual, real-time systems and may travel up to 15% for launch testing.

Qualifications

  • 4+ years of QA engineering or test automation experience.
  • Experience designing and running evals for LLM-based products.
  • Strong Python and/or TypeScript test automation skills.
  • Production experience with UI automation frameworks (Playwright, Cypress, Selenium, Appium).
  • Daily use of agentic coding tools (Claude Code, Cursor, Codex).
  • Solid understanding of LLM agents, tool use, memory, and failure modes.
  • Comfort with manual and exploratory testing, and judgement on automation needs.
  • High agency and ability to navigate ambiguity in fast-moving environments.
  • Strong communication to explain risks and results to stakeholders.
  • Care for shipping reliable, safe AI experiences.

Responsibilities

  • Design, build, and maintain AI evaluation suites for Napster agents and digital twins, ensuring accuracy, grounding, and safety.
  • Build golden datasets and test cases with customers and engineers; convert conversations into regression datasets.
  • Implement LLM-as-judge and rubric-based scoring; calibrate against human review and monitor quality metrics.
  • Develop UI test automation for web, mobile, and kiosk front ends; integrate tests into CI/CD pipelines.
  • Run manual exploratory and device testing for Napster hardware in real deployments; report bugs clearly.
  • Codify reusable testing templates and patterns; contribute to agent skills and plugins.

Skills

QA engineering
LLM eval design
Python
TypeScript
Playwright
Cypress
Selenium
Appium
Claude Code/Cursor/Codex
Manual & Exploratory testing
Adaptability
Communication
Quality mindset

Tools

Playwright
Cypress
Selenium
Appium
GitHub Actions

Job description

About Napster

Napster has always been about democratizing access: first to music in 1999, and today to expertise. We are a frontier AI company powering the next generation of embodied and agentic AI, building lifelike AI agents with voice, video presence, persistent memory, and real-time utility across apps, websites, physical spaces, and workplaces. Our platform, including the Napster Omniagent API, our real-time conversational video models, and hardware like Napster Station and Napster View, lets organizations deploy AI coworkers, digital twins, and specialist agents that engage people face to face. Our team is a fast-growing group of engineers, researchers, designers, and business leaders working together to make AI that is useful, trustworthy, and human in how it connects.

About the role

As a QA Engineer on the Professional Services team at Napster, you will own the quality of the AI agents and experiences we ship to our most strategic customers. Your core mission is to make quality measurable and automatic: writing AI evals that tell us whether an agent is accurate, on-brand, and safe, building UI test automation for the web, mobile, and kiosk experiences around it, and wiring both into the pipelines our Forward Deployed Engineers ship through every day.

Napster's products are hard to test with traditional methods alone. Our agents speak, see, remember, and use tools in real time, often in multiple languages and inside customer systems. You will design how we test that, whether it's a digital twin of an executive greeting shoppers, a multilingual agent in a public-sector kiosk, or a crew of specialist agents working inside enterprise workflows.

Automation comes first, but you will also roll up your sleeves for hands-on exploratory and acceptance testing when a deployment needs it, including on real hardware and, occasionally, on site. Working closely with Forward Deployed Engineers, Solution Engineering, Enablement, Product, and Engineering, you'll be the person who can say with evidence that a deployment is ready for production.

How we build

Our QA work is AI-native. You'll use agentic coding tools like Claude Code, Cursor, and Codex as everyday teammates to generate test cases, write and maintain automation, triage failures, and build eval datasets in hours rather than weeks. We treat tests and evals as product: versioned, reviewed, and packaged into reusable agent skills, MCP servers, and templates so every deployment starts with a stronger safety net than the last. If you've used an LLM to judge another LLM, have opinions about flaky tests, and believe a good eval suite is the fastest way to ship AI with confidence, you'll feel at home here.

Responsibilities
  • Design, build, and maintain AI evaluation suites for Napster agents and digital twins, covering answer accuracy, grounding in customer knowledge, tool and function-call correctness, persona and brand fidelity, conversation quality, multilingual behavior, and safety guardrails.
  • Build golden datasets and test cases with customers and FDEs, including adversarial, edge-case, and red-team prompts, and turn real conversation transcripts into regression sets.
  • Implement LLM-as-judge and rubric-based scoring, calibrate judges against human review, and track quality metrics over time so model, prompt, or knowledge changes never ship as silent regressions.
  • Build and own UI test automation for customer-facing experiences across web, mobile, and kiosk front ends, using frameworks such as Playwright, Cypress, or Appium.
  • Automate end-to-end and API tests for agent integrations, MCP servers, tool calls, and third-party systems, including latency and reliability checks for real-time voice and video sessions.
  • Integrate evals and automated tests into CI/CD so every deployment has clear, automated quality gates and readable reports for the team and the customer.
  • Run manual exploratory, acceptance, and device testing when needed, including on Napster Station and Napster View hardware and in physical installations, and write clear, reproducible bug reports.
  • Define test strategies and acceptance criteria with Solution Engineering and FDEs at the start of each engagement, and sign off on production readiness.
  • Codify repeatable testing patterns into reusable eval templates, test harnesses, agent skills, and plugins, and feed quality insights back to Product and Engineering.
  • Contribute to internal programs, including agent dogfooding and enablement, that help the wider team adopt AI-native testing practices.
  • Stay current on AI evaluation and testing tools, from eval frameworks and observability platforms to new approaches for testing voice, video, and agentic systems.
  • Travel to customer sites for launch testing and acceptance when needed, estimated at up to 15% depending on location.
  • Be a champion for quality, reliability, and responsible AI practices in everything we ship.
You may be a good fit if you have
  • 4+ years of experience in QA engineering, SDET, or test automation, ideally including customer-facing, consulting, or professional services work.
  • Hands-on experience designing and running evals for LLM-based products, such as golden datasets, rubric scoring, LLM-as-judge, and regression testing of prompts and agents.
  • Strong test automation skills in Python and/or TypeScript, with a track record of building maintainable UI and API test suites from scratch.
  • Production experience with UI automation frameworks such as Playwright, Cypress, Selenium, or Appium, and with integrating tests into CI/CD pipelines (GitHub Actions, GitLab CI, or similar).
  • Daily use of agentic coding tools such as Claude Code, Cursor, or Codex, and a clear point of view on how to use them to write, maintain, and debug tests.
  • Solid understanding of how LLM agents work, including tool use, retrieval, memory, and their typical failure modes like hallucinations, prompt injection, and inconsistent behavior.
  • Comfort with manual and exploratory testing, and the judgment to know when a human needs to look at something and when it should be automated.
  • High agency and comfort navigating ambiguity in fast-moving product and customer environments.
  • Strong communication skills to explain quality risks and test results to developers, customers, and executives, with a low ego and collaborative approach.
  • Genuine care about shipping AI experiences that are reliable, safe, and that people actually want to talk to.
Nice to have
  • Experience with LLM eval and observability tooling such as Langfuse, Braintrust, LangSmith, Arize, Promptfoo, or DeepEval.
  • Experience testing real-time voice, video, or multimodal systems (WebRTC, streaming audio/video, speech-to-text, text-to-speech, avatars), including latency and audio quality measurement.
  • Experience with AI red teaming, safety testing, or prompt-injection testing.
  • Experience testing hardware or physical deployments such as kiosks, venues, or retail spaces.
  • Experience with performance and load testing tools such as k6, Locust, or JMeter.
  • Experience building MCP servers, agent skills, or internal test tooling that other engineers use.
  • Testing multilingual products, especially Arabic and right-to-left interfaces, or experience working in the Middle East, Europe, or other international markets.
  • Background in retail, government and public sector, education, healthcare, or another enterprise vertical.
  • Familiarity with enterprise IT, security reviews, accessibility testing, and integration patterns (SSO, CRM, e-commerce, content management).
  • Public work that shows how you test: open-source eval suites or test tools, blog posts, demos, or talks.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Quality Assurance Engineer
Quality Assurance Engineer

Napster Corp. • United States

Remote
USD 90,000 - 120,000
QA Automation Engineer
QA Automation Engineer

GCS Recruitment • United States

Remote
USD 120,000 - 170,000
Senior Staff SDET (Customer Engineering)
Senior Staff SDET (Customer Engineering)

Hims & Hers Health, Inc. • United States

Remote
USD 180,000 - 240,000
Generous PTO
Full healthcare
401(k) with matching
+1
1214 | Senior Agentic QA Engineer
1214 | Senior Agentic QA Engineer

Intetics • United States

Remote
USD 150,000 - 210,000
1214 | Senior Agentic QA Engineer
1214 | Senior Agentic QA Engineer

Intetics • Naples (FL)

On-site
USD 120,000 - 170,000
AI Evaluation Engineer
AI Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
QA AI Automation Engineer
QA AI Automation Engineer

Dynasty Financial Partners • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
Quality Assurance Automation Engineer
Quality Assurance Automation Engineer

Structurely • United States

Remote
USD 120,000 - 180,000
Remote work
Senior AI Systems Quality Engineer
Senior AI Systems Quality Engineer

Abacus Insights • United States

On-site
USD 120,000 - 180,000
Senior QA Engineer
Senior QA Engineer

Vi • United States

Remote
USD 120,000 - 180,000