Senior AI Test Automation Engineer Frontline AI Quality

Genuine Parts Company

United States

On-site

USD 120,000 - 155,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

GPC seeks a Senior AI Test Automation Engineer to design, build, and maintain automated tests for traditional software and LLM-powered features. You will work directly with users and delivery teams to capture real-world usage and feed it into evolving evaluation frameworks, ensuring quality across the software delivery lifecycle.

You will partner with AI engineering, QA, and product teams to expand coverage, validate LLM behavior, and integrate automated tests and evaluations into CI/CD

Qualifications

  • 5+ years in test automation engineering or QA roles.
  • 1–2+ years testing or evaluating LLM-powered apps and datasets.
  • Hands-on LangSmith tracing, evaluation, and observability experience.
  • Experience with LangGraph graph workflows and tool calls.
  • Proficient in Playwright (TypeScript) and Python/TS for custom evaluators.

Responsibilities

  • Design and implement automated UI, API, integration, and regression tests for LLM-powered surfaces.
  • Triage failures, differentiate framework issues from defects, and manage test data provenance.
  • Integrate tests into CI/CD and gate changes with eval datasets and quality thresholds.
  • Collaborate with QA, development, product, and AI teams to raise coverage and maintainability.
  • Engage with QA CoE to align automation and AI evaluation standards.

Skills

LangSmith
LangGraph
Playwright
TypeScript
Python
CI/CD
Azure DevOps
Jenkins
Git
GitHub Actions
Agile/Scrum
UI Testing
API Testing

Tools

Playwright
Git
Azure DevOps
Jenkins
GitHub Actions
Cucumber

Job description

GPC seeks a Senior AI Test Automation Engineer to design, build, and maintain automated tests for traditional software and LLM-powered features. You will work directly with users and delivery teams to capture real-world usage and feed it into evolving evaluation frameworks, ensuring quality across the software delivery lifecycle.

You will partner with AI engineering, QA, and product teams to expand coverage, validate LLM behavior, and integrate automated tests and evaluations into CI/CD

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

QA Automation Engineer - GenAI / LLM Testing (Healthcare)
QA Automation Engineer - GenAI / LLM Testing (Healthcare)

Luxoft • United States

Remote
USD 110,000 - 150,000
Lead QA Engineer for AI-Driven Testing & Automation
Lead QA Engineer for AI-Driven Testing & Automation

Genpact Ltd. • United States

Remote
USD 60,000 - 75,000
GenAI QA Automation Engineer – LLM Testing
GenAI QA Automation Engineer – LLM Testing

Luxoft • United States

Remote
USD 110,000 - 150,000
Senior AI Test Engineer for GenAI & LLM Apps
Senior AI Test Engineer for GenAI & LLM Apps

Anaplan • North Carolina

Hybrid
USD 120,000 - 180,000
Senior .NET Test Automation Engineer
Senior .NET Test Automation Engineer

EPAM Systems • United States

Remote
USD 120,000 - 160,000
AI Quality Architect
AI Quality Architect

EPAM Systems • United States

Remote
USD 140,000 - 210,000
Software Engineer in Test (AI-Driven Functional and Automation)
Software Engineer in Test (AI-Driven Functional and Automation)

Groupe Kardol • Philadelphia

Hybrid
USD 130,000 - 180,000
Lead Software Test Engineer — AI-Driven Quality
Lead Software Test Engineer — AI-Driven Quality

Groupe Kardol • Lower Providence Township (PA)

On-site
USD 123,000 - 205,000
Flexible vacation
Seven paid holidays
Wellness program
+1
Senior Java Test Automation Engineer with AI Expertise
Senior Java Test Automation Engineer with AI Expertise

EPAM Systems • United States

Remote
USD 120,000 - 170,000
Lead QA Engineer - AI-Driven Testing & Automation (Remote)
Lead QA Engineer - AI-Driven Testing & Automation (Remote)

Genpact Ltd. • Arizona

Remote
USD 60,000 - 75,000