Senior AI Test Automation Engineer Frontline AI Quality

USA MOT Motion Industries, Inc.; Alton (IL); On-site; USD 120,000 - 155,000; Full time

Job summary

GPC seeks a Senior AI Test Automation Engineer to design, build, and maintain automated tests for traditional software and LLM-powered features. You will work directly with users and delivery teams to capture real-world usage and feed it into evolving evaluation frameworks, ensuring quality across the software delivery lifecycle.

You will partner with AI engineering, QA, and product teams to expand coverage, validate LLM behavior, and integrate automated tests and evaluations into CI/CD

Qualifications

• 5+ years in test automation engineering or QA roles.
• 1–2+ years testing or evaluating LLM-powered apps and datasets.
• Hands-on LangSmith tracing, evaluation, and observability experience.
• Experience with LangGraph graph workflows and tool calls.
• Proficient in Playwright (TypeScript) and Python/TS for custom evaluators.

Responsibilities

• Design and implement automated UI, API, integration, and regression tests for LLM-powered surfaces.
• Triage failures, differentiate framework issues from defects, and manage test data provenance.
• Integrate tests into CI/CD and gate changes with eval datasets and quality thresholds.
• Collaborate with QA, development, product, and AI teams to raise coverage and maintainability.
• Engage with QA CoE to align automation and AI evaluation standards.

Skills

LangSmith

LangGraph

Playwright

TypeScript

Python

CI/CD

Azure DevOps

Jenkins

Git

GitHub Actions

Agile/Scrum

UI Testing

API Testing

Tools

Playwright

Git

Azure DevOps

Jenkins

GitHub Actions

Cucumber

Job description

GPC seeks a Senior AI Test Automation Engineer to design, build, and maintain automated tests for traditional software and LLM-powered features. You will work directly with users and delivery teams to capture real-world usage and feed it into evolving evaluation frameworks, ensuring quality across the software delivery lifecycle.

You will partner with AI engineering, QA, and product teams to expand coverage, validate LLM behavior, and integrate automated tests and evaluations into CI/CD