Senior AI Test & Evaluation Engineer

Genuine Parts Company; Alabama; On-site; USD 120,000 - 150,000; Full time

Job summary

Genuine Parts Company is seeking a Senior AI Test Automation Engineer in Alabama to design, develop, and maintain automated testing for traditional software and LLM-powered applications. You will work closely with users and delivery teams to capture usage patterns and continuously expand the evaluation framework, ensuring robust QA across the software delivery lifecycle.

The role requires hands-on expertise with LangSmith and LangGraph, Playwright (TypeScript), Python/TypeScript, and strong

Qualifications

• 5+ years of experience in test automation engineering or QA
• 1–2+ years testing or evaluating LLM-powered apps, including eval datasets
• Hands-on experience with LangSmith for tracing, evaluation, and observability
• Hands-on experience with LangGraph graphs, workflows, and tool calls
• Experience building automated test suites with Playwright, preferably TS
• Proficiency in Python and/or TypeScript to author evaluators and test code
• Familiarity with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions) and integrating eval runs
• Working knowledge of Git and source control
• Understanding of Agile/Scrum delivery and ceremonies
• Ability to translate requirements/user feedback into test and eval coverage
• Strong problem-solving, communication, and collaboration, including direct user engagement

Responsibilities

• Design, develop, and maintain automated test scripts for UI, API, integration, and regression testing
• Integrate automated tests into CI/CD pipelines for fast feedback and quality validation
• Collaborate with QA, development, product, and business teams to map requirements to test coverage
• Analyze automation failures and distinguish framework issues from product defects
• Support test data management and environment stability for tests and datasets
• Report on test execution results, eval outcomes, and quality trends
• Participate in code reviews and contribute to automation standards and reusable components
• Align automation and AI evaluation practices with enterprise QA CoE and standards

Skills

LangSmith

LangGraph

Playwright

TypeScript

Python

CI/CD Tools

Git

Agile/Scrum

E2E testing

LLM eval

Tools

LangSmith

LangGraph

Playwright

Git

Azure DevOps

Jenkins

GitHub Actions

Python

TypeScript

Job description

Genuine Parts Company is seeking a Senior AI Test Automation Engineer in Alabama to design, develop, and maintain automated testing for traditional software and LLM-powered applications. You will work closely with users and delivery teams to capture usage patterns and continuously expand the evaluation framework, ensuring robust QA across the software delivery lifecycle.

The role requires hands-on expertise with LangSmith and LangGraph, Playwright (TypeScript), Python/TypeScript, and strong