AI/LLM Automation Test Engineer

Brilliant Infotech Inc.; Birmingham (AL); Hybrid; USD 110,000 - 140,000; Full time

Job summary

Brilliant Infotech Inc. in Birmingham, AL is seeking a Senior Quality Engineer / SDET with strong hands-on AI/LLM testing and test automation.

The role focuses on validating AI-powered applications, building evaluation frameworks to assess accuracy, groundedness, and hallucinations, and delivering reliable software through automated tests. You will develop tests with Playwright, TS, or Python, create LLM evaluation datasets, integrate tests into CI/CD, and work in a hybrid environment with

Qualifications

• 10+ years QA/SDET/test automation experience.
• 2–3+ years hands-on AI/LLM testing and evaluation.
• Experience with evaluation datasets, ground truth, LLM judges, scoring, and quality thresholds.
• Hands-on LangSmith and LangGraph/AI agent workflow experience.
• Strong Playwright with TypeScript and/or Python.
• API, integration, regression, and E2E testing experience.
• CI/CD, Git, and Agile development experience.

Responsibilities

• Build automated tests for AI/LLM apps using Playwright, TS and/or Python.
• Develop LLM evaluation datasets, ground truth, regression suites, and quality thresholds.
• Evaluate nondeterministic AI outputs with LLM-as-a-Judge and evaluators.
• Test RAG, AI agents, prompts, tool/API calls, and multi-step workflows.
• Use LangSmith for tracing, datasets, experiments, and evaluations, and LangGraph for agent testing.
• Analyze AI failures across model behavior, prompts, retrieval, and defects.
• Integrate AI evaluations and automated regression testing into CI/CD pipelines.
• Identify and resolve flaky tests and improve reliability.

Skills

QA/SDET

AI/LLM testing

Test automation

CI/CD

Agile

Git

API testing

E2E testing

Tools

Playwright

TypeScript

Python

LangSmith

LangGraph

CI/CD tools

Job description

Role: Senior Quality Engineer – AI/LLM Test Automation

Location: Birmingham, AL - Hybrid

Type of Employment: Full Time Permanent

Job Description:

We are seeking a Senior Quality Engineer / SDET with strong hands‑on experience in AI/LLM testing and test automation. The ideal candidate will have experience validating AI‑powered applications and building LLM evaluation frameworks to assess accuracy, relevance, groundedness, hallucinations, reliability, and consistency of AI‑generated responses.

Key Responsibilities:

• Build automated tests for AI/LLM applications using Playwright, TypeScript and/or Python.
• Develop LLM evaluation datasets, ground truth, regression suites, scoring criteria, and quality thresholds.
• Evaluate nondeterministic AI outputs using LLM-as-a-Judge, heuristic/code‑based evaluators, and comparison‑based evaluation.
• Test RAG, AI agents, prompts, tool/API calls, and multi‑step workflows.
• Use LangSmith for tracing, datasets, experiments, and evaluations, and LangGraph or similar frameworks for agent workflow testing.
• Analyze AI failures across model behavior, prompts, retrieval, and application defects.
• Integrate AI evaluations and automated regression testing into CI/CD pipelines.
• Identify and resolve flaky tests and continuously improve test reliability.

Must-Have Skills:

• 10+ years QA/SDET/Test Automation experience.
• 2–3+ years hands‑on AI/LLM testing and LLM evaluation.
• Experience with evaluation datasets, ground truth, LLM judges, scoring, and quality thresholds.
• Hands‑on LangSmith and LangGraph/AI agent workflow experience.
• Strong Playwright with TypeScript and/or Python.
• API, integration, regression, and E2E testing experience.
• CI/CD, Git, and Agile development experience.