Lead QA Engineer for AI/ML Testing & CI/CD Quality

7Seventy

United States

Hybrid

USD 140,000 - 165,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity

Job summary

7Seventy is seeking a Lead QA Engineer to build and formalize a QA function for a growing AI software environment. You will hands-on write test cases, develop AI evaluation frameworks, automate test coverage, and implement CI/CD quality gates while triaging production issues with Engineering.

You will work with Python, pytest, Playwright or Selenium, REST APIs, Azure, and SOC 2 Type II documentation to ensure high-quality releases in a remote US setting.

Qualifications

  • 7+ years of professional experience in Software QA Engineering.
  • At least 2 years of hands-on experience testing AI or machine learning systems.
  • Experience writing test cases against LLM outputs and evaluating non-deterministic AI behavior.
  • Proven experience building AI evaluation pipelines from scratch.
  • Experience building and maintaining automated test suites in production software environments.
  • Experience integrating QA quality gates into CI/CD pipelines and owning the process end to end.

Responsibilities

  • Design and implement evaluation frameworks for LLMs and agentic AI across environments.
  • Write assertions, define behavioral contracts, and establish regression baselines.
  • Develop evaluation approaches for non-deterministic AI systems where determinism is insufficient.
  • Apply distributions, confidence intervals, and advanced methods to AI outputs.
  • Build tooling to support reliable evaluation of AI-generated results, including complex outputs.
  • Write and maintain automated test suites for end-to-end, integration, and regression testing.
  • Own performance and load testing for latency-sensitive AI inference paths.
  • Enforce quality gates in CI/CD and participate in production bug triage.
  • Create reproducible test cases and establish regression baselines for model behavior.
  • Document SOC 2 Type II compliant QA artifacts and processes.

Skills

QA Engineering
AI/ML Testing
Python
CI/CD
API Testing
Regression Testing
Performance Testing
SOC 2

Tools

pytest
Playwright
Selenium
Azure
RAGAS
LangSmith
Langfuse

Job description

7Seventy is seeking a Lead QA Engineer to build and formalize a QA function for a growing AI software environment. You will hands-on write test cases, develop AI evaluation frameworks, automate test coverage, and implement CI/CD quality gates while triaging production issues with Engineering.

You will work with Python, pytest, Playwright or Selenium, REST APIs, Azure, and SOC 2 Type II documentation to ensure high-quality releases in a remote US setting.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI QA Engineer - CI/CD & SOC 2 Focus
Senior AI QA Engineer - CI/CD & SOC 2 Focus

7Seventy • United States

Hybrid
USD 110,000 - 135,000
Equity
Lead Software Development Engineer in Test - AI
Lead Software Development Engineer in Test - AI

HighLevel • Texas

Hybrid
USD 140,000 - 200,000
Azure Quality Engineering Lead | AI & Data Testing
Azure Quality Engineering Lead | AI & Data Testing

3Cloud • United States

Remote
USD 14,000 - 24,000
Quarterly bonus
Annual bonus
Remote work option
Remote QA Automation Lead: API+UI & AI-Driven Testing
Remote QA Automation Lead: API+UI & AI-Driven Testing

Juniper Square • United States

Remote
USD 140,000 - 190,000
QA Engineer
QA Engineer

7Seventy • United States

Hybrid
USD 110,000 - 135,000
Equity
Senior QA Engineer - Azure & AI Testing Lead
Senior QA Engineer - Azure & AI Testing Lead

3Cloud • United States

Remote
USD 100,000 - 140,000
Remote work
Competitive compensation
Bonus potential
Lead QA Automation Architect - Playwright & CI/CD Leader
Lead QA Automation Architect - Playwright & CI/CD Leader

Geak Minds • United States

Remote
USD 100,000 - 160,000
Lead QA Automation Engineer - Playwright & AI Testing
Lead QA Automation Engineer - Playwright & AI Testing

Robotic Process Automation LLC • Omaha (NE)

On-site
USD 120,000 - 160,000
AI Quality & Test Strategy Leader
AI Quality & Test Strategy Leader

HighLevel • Texas

Hybrid
USD 140,000 - 200,000
Lead QA Engineer — Remote Playwright & AI Testing
Lead QA Engineer — Remote Playwright & AI Testing

Appfire • United States

Remote
USD 120,000 - 180,000
Equity
25 PTO days
Fully remote within Bulgaria