Senior AI Test Automation Engineer

Jobtailor; Birmingham (AL); On-site; USD 90,000 - 130,000; Full time

Job summary

Jobtailor is seeking a senior QA automation engineer to join our team in the US, focusing on LangSmith‑driven evaluation and LangGraph workflows. You will own end‑to‑end test strategies for LLM-powered surfaces, build robust automated suites with Playwright, and drive reliability across CI/CD pipelines.

You will collaborate with QA, development, product, and business teams to translate requirements into automated coverage, conduct online evaluations, and help detect quality drift in production

Qualifications

• 5+ years of experience in test automation engineering, software quality assurance, or a related role.
• 1–2+ years of hands-on experience testing or evaluating LLM-powered applications.
• Hands-on experience with LangSmith for tracing, evaluation, and observability of LLM applications.
• Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling.
• Hands-on experience building and maintaining automated test suites using Playwright, preferably with TypeScript.
• Working proficiency in Python and/or TypeScript.
• Experience automating UI and API testing, and experience with API testing tools or frameworks.

Responsibilities

• Engage directly with users and stakeholders to collect feedback and identify consistent user flows through LLM-powered features.
• Translate feedback into evaluation datasets in LangSmith.
• Design, build, and maintain offline evaluation suites for regression, benchmarking, and backtesting.
• Develop and calibrate heuristic/code-based checks and LLM-as-judge evaluators.
• Validate evaluator reliability against human review.
• Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith.
• Manage annotation queues and human-in-the-loop feedback workflows.
• Analyze agent trajectories to identify failure points and distinguish LLM variance from defects.
• Integrate evaluation runs into CI/CD pipelines as quality gates.
• Support production auditing and quality drift detection for LLM applications.
• Design, develop, and implement automated UI, API, integration, regression tests for LLM surfaces.
• Collaborate with QA, development, product, and business teams to translate requirements into coverage.
• Analyze and triage automation test failures.
• Support test data management and test-environment stability.
• Report results, coverage, trends, and risks.
• Participate in code reviews and contribute to automation standards.
• Engage with QA Center of Excellence to align practices with enterprise standards.

Skills

Test Automation

LangSmith

LangGraph

Automated Tests

CI/CD

Python

TypeScript

BDD/Gherkin

API Testing

Playwright

Perf Testing

Accessibility

Security Testing

Tools

LangSmith

LangGraph

Playwright

Azure DevOps

Jenkins

GitHub Actions

Cucumber

Test Management Tools

Job description

• Engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify consistent user flows through LLM-powered features


• Translate user feedback and production traces into curated evaluation datasets in LangSmith


• Design, build, and maintain offline evaluation suites for regression, benchmarking, and backtesting


• Develop and calibrate heuristic/code-based checks, LLM-as-judge evaluators, and pairwise comparisons


• Validate evaluator reliability against human review


• Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith


• Manage annotation queues and human-in-the-loop feedback workflows


• Analyze agent trajectories and multi-step LangGraph executions to identify failure points and distinguish LLM variance from product defects


• Integrate evaluation runs and automated tests into CI/CD pipelines as quality gates


• Support production auditing, online evaluations, quality drift detection, and alerting


• Design, develop, and implement automated UI, API, integration, regression, and deterministic E2E tests for LLM-powered application surfaces


• Collaborate with QA, development, product, and business teams to translate requirements, acceptance criteria, and expected agent behaviors into test and evaluation coverage


• Analyze and triage automation test failures


• Support test data management and test-environment stability


• Report test results, evaluation outcomes, coverage, quality trends, and risks


• Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices


• Engage with the QA Center of Excellence to align automation and AI evaluation practices with enterprise standards

Requirements

• Must be eligible to work in the US without Visa Sponsorship


• 5+ years of experience in test automation engineering, software quality assurance, or a related role


• 1–2+ years of hands‑on experience testing or evaluating LLM-powered applications


• Hands‑on experience with LangSmith for tracing, evaluation, and observability of LLM applications


• Hands‑on experience with LangGraph, including graph‑based agent workflows, state and context management, and tool calling


• Hands‑on experience building and maintaining automated test suites using Playwright, preferably with TypeScript


• Working proficiency in Python and/or TypeScript


• Experience automating UI and API testing, and experience with API testing tools or frameworks


• Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent


• Working knowledge of Git


• Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies


• Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage


• Strong problem‑solving, troubleshooting, communication, and collaboration skills


• Ability to understand and document end‑to‑end business processes, user journeys, and operational workflows


• Experience with online evaluations, production monitoring, and quality drift detection for LLM applications


• Experience with agent trajectory evaluation, RAG evaluation, or guardrails validation


• Familiarity with prompt engineering and prompt versioning workflows


• Understanding of statistical approaches to nondeterministic testing


• Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework


• Experience with contract testing tools


• Familiarity with SAFe and Agile Release Train (ART) practices


• Experience with AI‑assisted engineering practices


• Experience with performance, accessibility, mobile, or security test automation


• Experience with test management and defect‑tracking tools, such as Azure DevOps

Core Competencies

Demonstrates expertise in test automation engineering and quality assurance for LLM-powered applications, with a strong focus on developing and maintaining automated test suites, analyzing user feedback, and ensuring product quality through rigorous evaluation practices.

Highest-signal resume keywords

• Test Automation Engineering


• LangSmith Experience


• LangGraph Workflows


• Automated Test Suites Development


• CI/CD Tools Familiarity

Hard Skills

• Test Automation Engineering


• LLM Application Evaluation


• Automated UI Testing


• API Testing


• Python Programming


• TypeScript Programming


• BDD/Gherkin Scenarios


• Statistical Testing Approaches


• Performance Test Automation


• Accessibility Test Automation

Soft Skills

• Problem‑Solving


• Troubleshooting


• Communication


• Collaboration

Industry Keywords

• Agile Practices


• Scrum


• Quality Drift Detection


• Human‑in‑the‑Loop Workflows


• Production Monitoring

Tools & Technologies

• LangSmith


• LangGraph
  
• Playwright


• Azure DevOps


• Jenkins


• GitHub Actions


• Cucumber


• Test Management Tools