Senior AI Test & Evaluation Engineer

Motion; Birmingham, Northern (AL, KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Motion is seeking a Senior AI Test Automation Engineer to design, develop, and maintain automated testing for traditional software and LLM-powered applications. You will work directly with users and delivery teams to capture real-world usage patterns, translate them into evaluation datasets, and expand end-to-end testing coverage.

You will integrate tests into CI/CD pipelines and collaborate across QA, development, product, and AI engineering to raise quality throughout the software lifecycle.

Qualifications

• 5+ years in test automation engineering or related QA role.
• 1-2+ years testing/evaluating LLM-powered apps, including eval datasets and nondeterminism handling.
• Hands-on LangSmith for tracing, evaluation, and observability of LLM apps.
• Experience with LangGraph workflows and tool calls.
• Experience building and maintaining automated test suites using Playwright (TypeScript).
• Proficiency in Python and/or TypeScript for custom evaluators and instrumentation.
• Familiar with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions).
• GIT version control knowledge; Agile/Scrum experience.
• Ability to translate requirements into test/eval coverage and communicate with end users in a forward-deployed setting.

Responsibilities

• Design, develop, and implement automated tests for UI, API, integration, and regression testing, including E2E coverage for LLM-powered surfaces.
• Integrate automated tests into CI/CD pipelines for timely feedback and quality validation.
• Collaborate with QA, development, product, and business teams to translate requirements into test/eval coverage.
• Analyze and triage automation failures, distinguishing framework issues from product defects and LLM nondeterminism.
• Support test data management, dataset versioning, and test-environment stability.
• Report test results, eval outcomes, and quality trends; participate in code reviews and standards.

Skills

Test automation engineering

QA / quality assurance

Python

TypeScript

Playwright

CI/CD

Git

Agile/Scrum

Requirements analysis

Communication

Tools

LangSmith

LangGraph

Playwright

Azure DevOps

Jenkins

GitHub Actions

Git

Job description

Motion is seeking a Senior AI Test Automation Engineer to design, develop, and maintain automated testing for traditional software and LLM-powered applications. You will work directly with users and delivery teams to capture real-world usage patterns, translate them into evaluation datasets, and expand end-to-end testing coverage.

You will integrate tests into CI/CD pipelines and collaborate across QA, development, product, and AI engineering to raise quality throughout the software lifecycle.