Founding AI Quality & Release Engineer

Wintermeyer Ventures; New York (NY); Remote; USD 170,000 - 200,000; Full time

Job summary

Wintermeyer Ventures is seeking a senior software quality engineer to own the full verification and release system for an AI/data startup. You will design agentic test harnesses, LLM graders, and CI pipelines, ensuring every build is trustworthy and releases are smooth.

With 7–10+ years in software or quality engineering and hands-on work with agentic AI, you will automate testing, improve gate controls, and drive reliable staging and production environments.

Qualifications

• 7 to 10+ years of software or quality engineering experience.
• 1 year hands-on building/testing evaluating agentic AI systems and LLM apps in the last year.
• Deep proficiency with GitHub Actions and GitHub CI/CD, including container builds and pipelines.
• Strong Python and TypeScript; comfortable with Docker, Kubernetes, and Helm.
• Familiarity with AWS, Snowflake, Databricks or Azure.
• Background in early-stage startups or modernizing QA/DevOps in larger orgs.
• Demonstrated ability to own systems end-to-end with minimal guidance.
• Authorized to work in the US or Canada without visa sponsorship now or in the future.

Responsibilities

• Design, build, and maintain agentic test harnesses and LLM-based evaluation systems that run tests automatically.
• Own regression coverage and release gates, stopping unsafe builds with evidence.
• Develop and improve CI pipelines, artifact publication workflows, and container build automation.
• Maintain reliable demo and staging environments that reflect production.
• Reduce bug-escape rates by improving automated signals rather than manual testing.
• Participate in business-hours incident escalations to reduce incidents over time.

Skills

GitHub Actions

Python

TypeScript

Docker

Kubernetes

Helm

AWS

Azure

Snowflake

Databricks

Job description

About the Role

This is the first dedicated quality-infrastructure hire at a well-backed AI/data startup, and it is a software engineering role through and through. You will own the full verification and release system, from agentic test harnesses and LLM graders to CI pipelines and deployment gates, making every green build trustworthy and every release uneventful.

What You’ll Do

• Design, build, and maintain agentic test harnesses and LLM-based evaluation systems that run and grade tests automatically.
• Own regression coverage and release gates, ensuring unsafe builds are stopped with evidence rather than intuition.
• Develop and improve CI pipelines, artifact publication workflows, and container build automation using GitHub Actions.
• Maintain reliable demo and staging environments that faithfully represent production.
• Drive down bug-escape rates by closing gaps in the automated signal, not by adding manual test runs.
• Participate in business-hours incident escalations and make incidents rarer and easier to diagnose over time.

What We’re Looking For

• 7 to 10 or more years of overall software or quality engineering experience.
• At least 1 year of hands-on experience building, testing, or evaluating agentic AI systems and LLM-based applications, applied within the last year.
• Deep proficiency with GitHub Actions and GitHub-based CI/CD, including container builds and pipeline automation.
• Strong command of Python and TypeScript; comfort with Docker, Kubernetes, and Helm.
• Familiarity with at least one major cloud or data platform such as AWS, Snowflake, Databricks, or Azure.
• Background in early-stage startups or in modernizing legacy QA and DevOps practices inside larger organizations.
• Demonstrated ability to take full ownership of systems and processes, make independent decisions, and execute without close guidance.
• Fully authorized to work in the US or Canada without visa sponsorship now or in the future.

Compensation & Benefits

Base salary range: $170,000 to $200,000 USD annually. Visa sponsorship is not available.

Location

Fully remote, open to candidates based anywhere in the United States or Canada. The company’s primary hub is New York, NY.