Senior SDET: ML/AI Quality & Automation Lead

HellaGoodAudio; Madison (WI); On-site; USD 153,000 - 229,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

401(k) matching

Paid time off

Wellness program

Employee discounts

Job summary

Sony Interactive Entertainment is hiring a Sr Software Engineer in Test focused on ML/AI quality to build automation for model evaluation and AI-powered workflows. You will lead quality strategy and collaborate with ML, engineering, and product teams.

Join a team that values collaboration and scalable automation, delivering reliable testing in CI/CD pipelines and across distributed systems, with a strong emphasis on non-deterministic AI behavior and risk management.

Qualifications

• Bachelor’s degree in Computer Science or equivalent.

Responsibilities

• Define and complete quality strategies, test plans, and automation coverage for ML-powered services.
• Use LLMs to generate, transform, and prioritize test cases for AI-powered experiences.
• Design scenario-based test suites for AI features, including adversarial prompts, edge cases, ambiguous inputs, and underrepresented user scenarios.
• Lead QE efforts for multi-functional projects, driving risk assessment, dependency management, and release readiness.
• Design, develop, and maintain scalable automation frameworks for backend services, APIs, and ML inference systems using Python and/or Java.
• Build automated validation for ML and LLM outputs, including ranking behavior, score distributions, prompt/response quality, hallucination indicators, and probabilistic model evaluation.
• Debug test failures, service anomalies, model inconsistencies, and AI behavior regressions to identify root causes and drive resolution.
• Perform functional, integration, regression, API, end-to-end, performance, and reliability testing for distributed systems.
• Improve automation reliability, reduce flakiness, and optimize execution efficiency.
• Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines and release workflows.
• Collaborate across teams to establish scalable quality standards, tooling, and guidelines.

Skills

Python

Java

LLMs

REST APIs

CI/CD

Distributed systems

Automation testing

Debugging

Education

Bachelor's degree in Computer Science or equivalent

Tools

pytest

JUnit

Selenium

Playwright

Cypress

Appium

Jenkins

GitHub Actions

AWS

GCP

Kubernetes

Docker

Job description

Sony Interactive Entertainment is hiring a Sr Software Engineer in Test focused on ML/AI quality to build automation for model evaluation and AI-powered workflows. You will lead quality strategy and collaborate with ML, engineering, and product teams.

Join a team that values collaboration and scalable automation, delivering reliable testing in CI/CD pipelines and across distributed systems, with a strong emphasis on non-deterministic AI behavior and risk management.