Senior AI/LLM QA & Test Automation Engineer

Brilliant Infotech Inc.; Birmingham (AL); Hybrid; USD 110,000 - 140,000; Full time

Job summary

Brilliant Infotech Inc. in Birmingham, AL is seeking a Senior Quality Engineer / SDET with strong hands-on AI/LLM testing and test automation.

The role focuses on validating AI-powered applications, building evaluation frameworks to assess accuracy, groundedness, and hallucinations, and delivering reliable software through automated tests. You will develop tests with Playwright, TS, or Python, create LLM evaluation datasets, integrate tests into CI/CD, and work in a hybrid environment with

Qualifications

• 10+ years QA/SDET/test automation experience.
• 2–3+ years hands-on AI/LLM testing and evaluation.
• Experience with evaluation datasets, ground truth, LLM judges, scoring, and quality thresholds.
• Hands-on LangSmith and LangGraph/AI agent workflow experience.
• Strong Playwright with TypeScript and/or Python.
• API, integration, regression, and E2E testing experience.
• CI/CD, Git, and Agile development experience.

Responsibilities

• Build automated tests for AI/LLM apps using Playwright, TS and/or Python.
• Develop LLM evaluation datasets, ground truth, regression suites, and quality thresholds.
• Evaluate nondeterministic AI outputs with LLM-as-a-Judge and evaluators.
• Test RAG, AI agents, prompts, tool/API calls, and multi-step workflows.
• Use LangSmith for tracing, datasets, experiments, and evaluations, and LangGraph for agent testing.
• Analyze AI failures across model behavior, prompts, retrieval, and defects.
• Integrate AI evaluations and automated regression testing into CI/CD pipelines.
• Identify and resolve flaky tests and improve reliability.

Skills

QA/SDET

AI/LLM testing

Test automation

CI/CD

Agile

Git

API testing

E2E testing

Tools

Playwright

TypeScript

Python

LangSmith

LangGraph

CI/CD tools

Job description

Brilliant Infotech Inc. in Birmingham, AL is seeking a Senior Quality Engineer / SDET with strong hands-on AI/LLM testing and test automation.

The role focuses on validating AI-powered applications, building evaluation frameworks to assess accuracy, groundedness, and hallucinations, and delivering reliable software through automated tests. You will develop tests with Playwright, TS, or Python, create LLM evaluation datasets, integrate tests into CI/CD, and work in a hybrid environment with