Senior AI Test Automation Engineer

Genuine Parts Company; Alabama; On-site; USD 120,000 - 150,000; Full time

Job summary

Genuine Parts Company is seeking a Senior AI Test Automation Engineer in Alabama to design, develop, and maintain automated testing for traditional software and LLM-powered applications. You will work closely with users and delivery teams to capture usage patterns and continuously expand the evaluation framework, ensuring robust QA across the software delivery lifecycle.

The role requires hands-on expertise with LangSmith and LangGraph, Playwright (TypeScript), Python/TypeScript, and strong

Qualifications

• 5+ years of experience in test automation engineering or QA
• 1–2+ years testing or evaluating LLM-powered apps, including eval datasets
• Hands-on experience with LangSmith for tracing, evaluation, and observability
• Hands-on experience with LangGraph graphs, workflows, and tool calls
• Experience building automated test suites with Playwright, preferably TS
• Proficiency in Python and/or TypeScript to author evaluators and test code
• Familiarity with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions) and integrating eval runs
• Working knowledge of Git and source control
• Understanding of Agile/Scrum delivery and ceremonies
• Ability to translate requirements/user feedback into test and eval coverage
• Strong problem-solving, communication, and collaboration, including direct user engagement

Responsibilities

• Design, develop, and maintain automated test scripts for UI, API, integration, and regression testing
• Integrate automated tests into CI/CD pipelines for fast feedback and quality validation
• Collaborate with QA, development, product, and business teams to map requirements to test coverage
• Analyze automation failures and distinguish framework issues from product defects
• Support test data management and environment stability for tests and datasets
• Report on test execution results, eval outcomes, and quality trends
• Participate in code reviews and contribute to automation standards and reusable components
• Align automation and AI evaluation practices with enterprise QA CoE and standards

Skills

LangSmith

LangGraph

Playwright

TypeScript

Python

CI/CD Tools

Git

Agile/Scrum

E2E testing

LLM eval

Tools

LangSmith

LangGraph

Playwright

Git

Azure DevOps

Jenkins

GitHub Actions

Python

TypeScript

Job description

Senior AI Test Automation Engineer

Summary

The Senior AI Test Automation Engineer designs, develops, maintains, and executes automated testing and evaluation solutions for both traditional software and LLM-powered applications. Operating in a forward-deployed capacity, this role works directly with users and delivery teams to capture real-world usage patterns and feedback, translating them into a continuously growing evaluation framework that validates LLM behavior against the consistent flows users actually follow. This role partners with delivery teams, QA, development, product, AI engineering, and the QA Center of Excellence (CoE) to establish scalable automation and evaluation practices, increase test and eval coverage, and integrate quality controls throughout the software delivery lifecycle, from pre-deployment regression testing through production observability.

You must be eligible to work in the US without Visa Sponsorship.

Responsibilities

LLM Evaluation & Observability (Core)

• Act as a forward-deployed quality engineer: engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify the consistent flows users follow through LLM-powered features.

• Translate user feedback and production traces into curated evaluation datasets in LangSmith, and continuously expand the eval framework as new feedback, edge cases, and failure modes are discovered.

• Design, build, and maintain offline evaluation suites (regression, benchmarking, and backtesting) that gate prompt, model, and LangGraph workflow changes before deployment.

• Develop and calibrate evaluators , heuristic/code-based checks, LLM-as-judge evaluators, and pairwise comparisons , and validate judge reliability against human review.

• Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith, ensuring trace coverage, quality, and useful metadata for debugging and analysis.

• Manage annotation queues and human-in-the-loop feedback workflows, routing interesting or problematic production runs to reviewers and feeding results back into datasets and evaluator calibration.

• Analyze agent trajectories and multi-step LangGraph executions (tool calls, state transitions, retrieval steps) to pinpoint failure points and distinguish nondeterministic LLM variance from genuine product defects.

• Integrate eval runs into CI/CD pipelines so that dataset versions, experiments, and quality thresholds provide automated feedback on every relevant change.

• Support the adoption of production auditing and monitoring capabilities — such as online evaluations on live traffic, quality drift detection, and alerting — to help teams detect issues in production (supportive to the role, not its core focus).

Test Automation (Core)

• Design, develop, and implement automated test scripts for UI, API, integration, and regression testing, including deterministic E2E coverage of LLM-powered application surfaces.

• Integrate automated tests into CI/CD pipelines to enable timely feedback and continuous quality validation.

• Collaborate with QA, development, product, and business teams to translate requirements, acceptance criteria, and expected agent behaviors into effective automated test and eval coverage.

• Analyze and triage automation test failures, differentiating framework or script issues from valid product defects , including the added dimension of expected LLM nondeterminism.

• Support test data management (including eval dataset versioning, splits, and provenance) and help identify or resolve test-environment stability issues.

• Report on test execution results, eval experiment outcomes, automation and eval coverage, quality trends, and risks.

• Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices.

• Engage with the QA CoE to align automation and AI evaluation practices with enterprise standards while contributing domain-specific feedback, lessons learned, and continuous-improvement opportunities.

Required Experience

• 5+ years of experience in test automation engineering, software quality assurance, or a related role.

• 1–2+ years of hands‑on experience testing or evaluating LLM-powered applications, including building eval datasets, defining pass/fail criteria for nondeterministic outputs, and using LLM-as-judge or heuristic evaluators.

• Hands‑on experience with LangSmith for tracing, evaluation, and observability of LLM applications , including creating datasets, running experiments, and configuring evaluators.

• Hands‑on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling.

• Hands‑on experience building and maintaining automated test suites using Playwright, preferably with TypeScript.

• Working proficiency in Python and/or TypeScript sufficient to author custom evaluators, tracing instrumentation, and test code.

• Experience automating UI and API testing, and experience with API testing tools or frameworks.

• Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent, including integrating eval runs as pipeline gates.

• Working knowledge of source-code version control, including Git.

• Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies.

• Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage.

• Strong problem‑solving, troubleshooting, communication, and collaboration skills , including comfort engaging directly with end users to gather feedback in a forward-deployed capacity.

Preferred / Nice to Have

• Ability to understand and document end-to-end business processes, user journeys, and operational workflows.

• Partner with business stakeholders and end users to translate process requirements into test scenarios, acceptance criteria, and LLM evaluation datasets.

• Identify process exceptions, edge cases, dependencies, and risks that may affect application or agent behavior.

• Validate that automated workflows and LLM-powered features produce outcomes aligned with defined business rules and user needs.

• Use production feedback and observed user behavior to continuously refine process coverage, test automation, and evaluation frameworks.

• Experience with online evaluations, production monitoring, and quality drift detection for LLM applications.

• Experience with agent trajectory evaluation, RAG evaluation (retrieval relevance, groundedness, hallucination detection), or guardrails validation.

• Familiarity with prompt engineering and prompt versioning workflows, and evaluating the impact of prompt or model changes.

• Understanding of statistical approaches to nondeterministic testing (multiple-run sampling, confidence thresholds, summary metrics across datasets).

• Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework.

• Experience with contract testing tools.

• Familiarity with SAFe and Agile Release Train (ART) practices.

• Experience with AI-assisted engineering practices (e.g., using AI coding agents to accelerate test and eval development).

• Experience with performance, accessibility, mobile, or security test automation.

• Experience with test management and defect-tracking tools, such as Azure DevOps.

GPC conducts its business without regard to sex, race, creed, color, religion, marital status, national origin, citizenship status, age, pregnancy, sexual orientation, gender identity or expression, genetic information, disability, military status, status as a veteran, or any other protected characteristic. GPC's policy is to recruit, hire, train, promote, assign, transfer and terminate employees based on their own ability, achievement, experience and conduct and other legitimate business reasons.