Senior AI Quality & Evaluation Engineer

Jobtailor; Birmingham (AL); On-site; USD 90,000 - 130,000; Full time

Job summary

Jobtailor is seeking a senior QA automation engineer to join our team in the US, focusing on LangSmith‑driven evaluation and LangGraph workflows. You will own end‑to‑end test strategies for LLM-powered surfaces, build robust automated suites with Playwright, and drive reliability across CI/CD pipelines.

You will collaborate with QA, development, product, and business teams to translate requirements into automated coverage, conduct online evaluations, and help detect quality drift in production

Qualifications

• 5+ years of experience in test automation engineering, software quality assurance, or a related role.
• 1–2+ years of hands-on experience testing or evaluating LLM-powered applications.
• Hands-on experience with LangSmith for tracing, evaluation, and observability of LLM applications.
• Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling.
• Hands-on experience building and maintaining automated test suites using Playwright, preferably with TypeScript.
• Working proficiency in Python and/or TypeScript.
• Experience automating UI and API testing, and experience with API testing tools or frameworks.

Responsibilities

• Engage directly with users and stakeholders to collect feedback and identify consistent user flows through LLM-powered features.
• Translate feedback into evaluation datasets in LangSmith.
• Design, build, and maintain offline evaluation suites for regression, benchmarking, and backtesting.
• Develop and calibrate heuristic/code-based checks and LLM-as-judge evaluators.
• Validate evaluator reliability against human review.
• Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith.
• Manage annotation queues and human-in-the-loop feedback workflows.
• Analyze agent trajectories to identify failure points and distinguish LLM variance from defects.
• Integrate evaluation runs into CI/CD pipelines as quality gates.
• Support production auditing and quality drift detection for LLM applications.
• Design, develop, and implement automated UI, API, integration, regression tests for LLM surfaces.
• Collaborate with QA, development, product, and business teams to translate requirements into coverage.
• Analyze and triage automation test failures.
• Support test data management and test-environment stability.
• Report results, coverage, trends, and risks.
• Participate in code reviews and contribute to automation standards.
• Engage with QA Center of Excellence to align practices with enterprise standards.

Skills

Test Automation

LangSmith

LangGraph

Automated Tests

CI/CD

Python

TypeScript

BDD/Gherkin

API Testing

Playwright

Perf Testing

Accessibility

Security Testing

Tools

LangSmith

LangGraph

Playwright

Azure DevOps

Jenkins

GitHub Actions

Cucumber

Test Management Tools

Job description

Jobtailor is seeking a senior QA automation engineer to join our team in the US, focusing on LangSmith‑driven evaluation and LangGraph workflows. You will own end‑to‑end test strategies for LLM-powered surfaces, build robust automated suites with Playwright, and drive reliability across CI/CD pipelines.

You will collaborate with QA, development, product, and business teams to translate requirements into automated coverage, conduct online evaluations, and help detect quality drift in production