QA Engineer - Agentic Systems

Meet Life Sciences; New York (NY); On-site; USD 110,000 - 170,000; Full time

Job summary

Meet Life Sciences seeks a senior quality and test automation engineer to lead QA for probabilistic LLMs and agentic workflows. You will own evaluation pipelines, CI/CD gates, and regression testing in a fast-paced startup.

In this ground-floor role, you will shape the quality function, design novel test strategies, and ensure the safety and reliability of enterprise platforms backed by top investors.

Qualifications

• 1 to 5 years of professional software engineering and testing experience with Python.
• Experience with end-to-end and UI test automation using Playwright.
• Experience or strong interest in testing non-deterministic ML/LLM systems.
• Ability to translate agent behaviors into regression tests.

Responsibilities

• Design LLM eval systems and evaluation pipelines.
• Establish CI/CD quality gates for AI software.
• Build end-to-end UI test suites with Playwright.
• Perform hands-on exploratory testing and defect analysis.

Skills

Python

CI/CD pipelines

Exploratory testing

End-to-end testing

ML/LLM testing

Tools

Playwright

Job description

Build the quality and evaluation engine for next-generation, compliant AI systems! If you are a senior quality and test automation engineer with 1 to 5 years of experience, eager to pioneer testing frameworks for probabilistic LLMs and agentic workflows, our client wants to hear from you. This ground-floor role makes you the first dedicated QA leader in a high-velocity startup. Step into an environment backed by top-tier investors where your evaluation pipelines, CI/CD gates, and rigorous testing standards will directly ensure the safety and reliability of life-changing enterprise platforms.

Description

Our client is a startup redefining pharmaceutical marketing with compliant, reference-backed automation, partnering with top life sciences manufacturers and dozens of major brands. In this early-stage environment, you'll step in as the foundational owner of software quality and AI evaluation.

• The Challenge: Build novel evaluation frameworks and automated test suites for non-deterministic AI agents where traditional pass/fail assertions do not apply.
• The Difference: Solve cutting-edge technical problems at the intersection of machine learning, compliance, and product reliability in a fast-growing startup.
• Core Responsibilities: Design LLM eval systems, establish CI/CD quality gates, build end-to-end UI suites with Playwright, and execute hands-on exploratory testing.
• Why Apply: Enjoy ground-floor equity, incredible ownership over the quality function, and the opportunity to define how reliable AI systems are built.

Skills

• 1 to 5 years of professional software engineering and testing experience, featuring strong Python skills and real-world implementation of CI/CD pipelines.
• Demonstrated expertise in end-to-end and UI test automation using Playwright, paired with a rigorous manual and exploratory QA discipline.
• Proven experience—or a deep appetite—for testing non-deterministic, ML, or LLM-based systems using golden datasets, LLM-as-judge, and statistical variance analysis.
• Strong instinct for error analysis, tracing system failures, and translating complex agent behaviors into permanent regression tests.