Quality Engineer (AI/LLM Test Automation)

Brilliant Infotech Inc.; Birmingham (AL); On-site; USD 110,000 - 150,000; Full time

Job summary

Brilliant Infotech Inc. in Birmingham, AL seeks a Senior Quality Engineer to lead AI/LLM test automation. You will build automated testing and evaluation frameworks for AI-powered apps, focusing on accuracy, relevance, and reliability of responses.

The role emphasizes nondeterministic output evaluation and regression testing. You will implement tests with Playwright (TypeScript) or Python, manage evaluation datasets, define ground truth, and integrate tests into CI/CD pipelines, ensuring robust

Qualifications

• 5+ years of QA / SDET / Test Automation experience.
• 1-2+ years hands-on testing of AI/LLM apps.
• Hands-on experience with LLM evaluation / AI evaluation.
• Experience with evaluation datasets and ground truth.
• Understanding of LLM judges / AI-as-a-judge and scoring.
• Experience testing nondeterministic AI outputs.
• LangSmith experience.
• LangGraph/agent workflow experience.
• Strong Playwright experience, preferably with TypeScript.
• Strong Python and/or TypeScript skills.
• API, integration, regression, and end-to-end testing.
• Experience CI/CD integration.

Responsibilities

• Design and implement automated tests for AI/LLM-powered apps using Playwright and TS/Python.
• Create and maintain LLM evaluation datasets and regression suites.
• Define ground truth, scoring criteria, and pass thresholds for AI outputs.
• Evaluate nondeterministic LLM results with evaluators and comparison methods.
• Validate accuracy, relevance, groundedness, hallucinations, and consistency of AI responses.
• Test AI workflows including prompts, tools, and multi-step flows.
• Trace/test AI behavior with LangSmith and similar frameworks.
• Investigate failures to determine model vs data vs product defects.
• Build automated regression checks and CI/CD integration.
• Improve test reliability and fix flaky tests.

Skills

Playwright

TypeScript

Python

LangSmith

LangGraph

AI/LLM testing

SDET

CI/CD

Git

Agile

Tools

Playwright

LangSmith

LangGraph

GitHub Copilot

Jenkins/GitHub Actions

Job description

Senior Quality Engineer AI/LLM Test Automation

Location: Birmingham, AL

Type: FTE

About the Role

We are looking for a Senior Quality Engineer / SDET with strong hands-on experience in AI/LLM testing and test automation. This role focuses on validating AI-powered applications where traditional pass/fail testing is not enough.

You will build automated testing and LLM evaluation frameworks to determine whether AI-generated responses are accurate, reliable, relevant, and production-ready.

What You'll Do

• Design and build automated tests for AI/LLM-powered applications using Playwright and TypeScript/Python.
• Create and maintain LLM evaluation datasets and regression test suites.
• Define ground truth, expected behavior, scoring criteria, and pass/fail thresholds for AI-generated outputs.
• Evaluate nondeterministic LLM responses using heuristic/code-based evaluators, AI/LLM judges, and comparison-based evaluation.
• Validate accuracy, semantic correctness, relevance, groundedness, hallucinations, and consistency of AI responses.
• Test AI workflows including RAG, agents, prompts, tool/API calls, and multi-step workflows.
• Use LangSmith for tracing, datasets, experiments, evaluation, and analyzing AI behavior.
• Work with LangGraph or similar agent frameworks to understand and test agent workflows and execution paths.
• Investigate failures and determine whether an issue is caused by LLM/model behavior, prompt changes, retrieval, or an actual product defect.
• Build automated regression checks and integrate AI evaluations into CI/CD pipelines.
• Identify and fix flaky automated tests and continuously improve test reliability.
• Use AI tools such as Claude, GitHub Copilot, Cursor, or similar tools to accelerate test creation, debugging, and failure analysis.

Must Have

• 5+ years of QA / SDET / Test Automation experience.
• 1 2+ years of hands-on experience testing AI/LLM applications.
• Hands-on experience with LLM evaluation / AI evaluation.
• Experience creating or working with evaluation datasets and ground truth.
• Understanding of LLM judges / AI-as-a-judge, evaluators, scoring, and quality thresholds.
• Experience testing nondeterministic AI-generated outputs.
• Hands-on LangSmith experience.
• Hands-on LangGraph or agent workflow experience.
• Strong Playwright experience, preferably with TypeScript.
• Strong Python and/or TypeScript skills.
• API, integration, regression, and end-to-end testing experience.
• Experience integrating automated tests into CI/CD.
• Strong understanding of Git and Agile development.