AI-Native QA Engineer for LLM/Agent Quality

Newton Research; Massachusetts; On-site; USD 115,000 - 130,000; Full time

Benefits offered by this job

Salary equity

Job summary

Newton Research is seeking a QA Engineer focused on AI-native quality in Boston/Needham, MA. You will own curated evals for agent behavior, validate prompts and skills, and ensure non-deterministic outputs are rigorously tested across the CI/CD pipeline.

You’ll collaborate with a release-readiness mindset, design guardrails, and drive regression testing for AI-driven features, with a strong emphasis on observability and reproducible results.

Qualifications

• 4+ years in QA or test engineering on a complex web product, ideally B2B SaaS shipping frequently
• Hands‑on experience building evals for LLM or agent products (datasets, rubrics, LLM‑as-judge, regression tracking)
• Working knowledge of how agents work: prompting, skills, tool calling, context, RAG and orchestration
• An automation-first instinct: you can show what you removed from a manual process
• Python (or similar) for eval tooling and data checks; Playwright or similar for end‑to‑end tests
• Statistical literacy: pass rates, variance and sample size when outputs are not deterministic
• Daily user of AI coding and testing assistants, with the judgment to verify their output
• Strong exploratory instincts and excellent written communication
• Nice to have: adtech, martech or marketing analytics domain; SSO / SAML / OAuth flows; data‑connector testing; eval or observability tooling

Responsibilities

• Own the eval suite for Newton's agents
• Test skill and prompt changes before they ship
• Cover the agent flows end to end
• Handle non-determinism with rigor
• Turn production and customer signal into evals
• Probe AI-specific risk
• Automate before you repeat
• Design agentic QA workflows in CI
• Keep human-judgment work sharp
• Write bug reports that are machine- and human-readable

Skills

QA

Eval tooling

Automation

Python

Playwright

Statistics

Documentation

Written communication

Tools

Playwright

Python

Job description

Newton Research is seeking a QA Engineer focused on AI-native quality in Boston/Needham, MA. You will own curated evals for agent behavior, validate prompts and skills, and ensure non-deterministic outputs are rigorously tested across the CI/CD pipeline.

You’ll collaborate with a release-readiness mindset, design guardrails, and drive regression testing for AI-driven features, with a strong emphasis on observability and reproducible results.