Forward-Deployed AI Test & Evaluation Engineer

Motion Industries; Birmingham (AL); On-site; USD 110,000 - 150,000; Full time

Job summary

Motion Industries in Birmingham, AL, seeks a Senior AI Test Automation Engineer to design, develop, and maintain automated testing for traditional software and LLM-powered apps. You will work directly with users and delivery teams to capture real-world usage and feed it into eval and automation frameworks, collaborating across QA, development, product, and AI engineering.

You must have 5+ years in test automation or QA, with 1–2+ years evaluating LLM-powered apps, LangSmith tracing, LangGraph

Qualifications

• 5+ years in test automation engineering or QA roles.
• 1-2+ years testing LLM-powered apps and eval datasets.
• Experience with LangSmith for tracing, evaluation, and observability.
• Hands-on with LangGraph workflows and tool calls.
• Proficient in Python and/or TypeScript for test code and evaluators.
• Familiarity with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions).
• Experience automating UI and API testing; version control with Git.
• Knowledge of Agile/Scrum and collaboration with cross-functional teams.

Responsibilities

• Design, implement, and maintain automated tests for UI, API, integration, and regression coverage for LLM-powered apps.
• Integrate automated tests into CI/CD pipelines for rapid feedback.
• Collaborate with QA, product, and engineering to translate requirements into robust test coverage.
• Analyze failures, differentiate framework issues from product defects, and address nondeterminism in LLM outputs.
• Support test data management and environment stability; report results and trends.
• Contribute to automation standards, reusable components, and best practices.

Skills

Playwright

Python

TypeScript

CI/CD

Git

Azure DevOps

LangSmith

LangGraph

QA

Automation

Tools

LangSmith

LangGraph

Playwright

Git

Azure DevOps

Job description

Motion Industries in Birmingham, AL, seeks a Senior AI Test Automation Engineer to design, develop, and maintain automated testing for traditional software and LLM-powered apps. You will work directly with users and delivery teams to capture real-world usage and feed it into eval and automation frameworks, collaborating across QA, development, product, and AI engineering.

You must have 5+ years in test automation or QA, with 1–2+ years evaluating LLM-powered apps, LangSmith tracing, LangGraph