Senior AI Quality Engineer — Remote Testing & Automation

Lever, Inc.

Town of Italy (NY)

Remote

USD 120,000 - 231,000

Full time

2 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Benefits offered by this job

Remote work

Home office budget

Flexible time off

Equity via VSOP

Job summary

Lever, Inc. is seeking a Senior Software Engineer, Quality for a partner-led, fully remote role based in Italy.

You will design, build, and operate AI-powered testing agents spanning test creation, execution, maintenance, evaluation, and failure triage, blending hands-on software engineering with test infrastructure and CI at scale. You'll shape testing patterns, tooling, and enablement practices, enable teams to own quality, and influence technical direction across a distributed engineering

Qualifications

5+ years of professional software engineering experience including production systems.
Hands-on experience with test infrastructure, developer tooling, or internal platforms.
Practical experience with LLMs/agent frameworks and evaluating AI outputs.
Distributed systems and CI/CD at scale, with attention to concurrency and reliability.
Ability to influence teams and drive tool adoption without direct authority.
Deep product understanding to build quality solutions in collaboration with engineers.
Strong judgment balancing quality, maintainability, and delivery speed.
Excellent collaboration in a distributed, cross-functional environment.
Experience with LLM evaluation, open-source tooling, or platform engineering preferred.
Experience with Java, TypeScript, or Python; Kubernetes and Helm advantageous.

Responsibilities

Build and operate AI agents across testing lifecycle: design, generation, maintenance, execution, triage.
Own infrastructure for dependable agent-driven testing: environments, data, interfaces, guardrails.
Develop reliable evaluation methods for agent output to reduce manual review.
Apply AI-driven evaluation to non-deterministic AI capabilities of the platform.
Collaborate with product engineering to keep testing tools relevant to new features.
Develop reusable testing patterns, tooling, and enablement practices for engineers.
Drive quality tooling adoption across teams without direct reporting lines.
Contribute to system design and continuous improvement of testing infrastructure.