Senior AI QA Engineer - CI/CD & SOC 2 Focus

7Seventy; United States; Hybrid; USD 110,000 - 135,000; Full time

Benefits offered by this job

Equity

Job summary

7Seventy is seeking an experienced QA Engineer to build a formal QA function for a growing AI software environment. You will be a hands-on contributor writing test cases, building AI evaluation frameworks, and developing automated coverage across AI inference, APIs, and cloud infrastructure.

You will triage production issues, establish regression baselines, and ensure SOC 2–type II readiness for artifacts and documentation.

Qualifications

• 7+ years of professional experience in Software QA Engineering.
• At least 2 years of hands-on testing AI or machine learning systems.
• Experience writing test cases for LLM outputs and evaluating non-deterministic AI behavior.
• Proven experience building AI evaluation pipelines from scratch.
• Experience building and maintaining automated test suites in production software environments.
• Experience integrating QA gates into CI/CD pipelines and owning the process end to end.

Responsibilities

• Design and execute evaluation frameworks for LLM and agentic AI outputs across development environments and client-deployed instances.
• Write assertions, define behavioral contracts, and establish regression baselines for model behavior.
• Develop evaluation approaches for non-deterministic AI systems where traditional deterministic assertions are insufficient.
• Apply statistical concepts such as distributions and confidence intervals when evaluating AI behavior.
• Build tooling that supports reliable evaluation of complex AI-generated outputs.
• Write and maintain automated test suites covering end-to-end, integration, and regression scenarios.
• Own performance and load testing for latency-sensitive AI inference paths.
• Implement and enforce quality gates within CI/CD pipelines.
• Participate in production bug triage and investigate failures across application and infrastructure layers.
• Build regression tests that prevent recurring production issues.

Skills

Python

Test automation

CI/CD quality gates

AI/ML testing

REST APIs

Tools

pytest

Playwright

Selenium

Job description

7Seventy is seeking an experienced QA Engineer to build a formal QA function for a growing AI software environment. You will be a hands-on contributor writing test cases, building AI evaluation frameworks, and developing automated coverage across AI inference, APIs, and cloud infrastructure.

You will triage production issues, establish regression baselines, and ensure SOC 2–type II readiness for artifacts and documentation.