Lead QA Engineer for AI/ML Testing & CI/CD Quality

7Seventy; Northern (KY); Hybrid; USD 140,000 - 165,000; Full time

Benefits offered by this job

Equity

Job summary

7Seventy is seeking a Lead QA Engineer to build and formalize a QA function for a growing AI software environment. You will hands-on write test cases, develop AI evaluation frameworks, automate test coverage, and implement CI/CD quality gates while triaging production issues with Engineering.

You will work with Python, pytest, Playwright or Selenium, REST APIs, Azure, and SOC 2 Type II documentation to ensure high-quality releases in a remote US setting.

Qualifications

• 7+ years of professional experience in Software QA Engineering.
• At least 2 years of hands-on experience testing AI or machine learning systems.
• Experience writing test cases against LLM outputs and evaluating non-deterministic AI behavior.
• Proven experience building AI evaluation pipelines from scratch.
• Experience building and maintaining automated test suites in production software environments.
• Experience integrating QA quality gates into CI/CD pipelines and owning the process end to end.

Responsibilities

• Design and implement evaluation frameworks for LLMs and agentic AI across environments.
• Write assertions, define behavioral contracts, and establish regression baselines.
• Develop evaluation approaches for non-deterministic AI systems where determinism is insufficient.
• Apply distributions, confidence intervals, and advanced methods to AI outputs.
• Build tooling to support reliable evaluation of AI-generated results, including complex outputs.
• Write and maintain automated test suites for end-to-end, integration, and regression testing.
• Own performance and load testing for latency-sensitive AI inference paths.
• Enforce quality gates in CI/CD and participate in production bug triage.
• Create reproducible test cases and establish regression baselines for model behavior.
• Document SOC 2 Type II compliant QA artifacts and processes.

Skills

QA Engineering

AI/ML Testing

Python

CI/CD

API Testing

Regression Testing

Performance Testing

SOC 2

Tools

pytest

Playwright

Selenium

Azure

RAGAS

LangSmith

Langfuse

Job description

7Seventy is seeking a Lead QA Engineer to build and formalize a QA function for a growing AI software environment. You will hands-on write test cases, develop AI evaluation frameworks, automate test coverage, and implement CI/CD quality gates while triaging production issues with Engineering.

You will work with Python, pytest, Playwright or Selenium, REST APIs, Azure, and SOC 2 Type II documentation to ensure high-quality releases in a remote US setting.