Senior AI/LLM QA Engineer - Test Automation

Brilliant Infotech Inc.; Birmingham (AL); On-site; USD 110,000 - 150,000; Full time

Job summary

Brilliant Infotech Inc. in Birmingham, AL seeks a Senior Quality Engineer to lead AI/LLM test automation. You will build automated testing and evaluation frameworks for AI-powered apps, focusing on accuracy, relevance, and reliability of responses.

The role emphasizes nondeterministic output evaluation and regression testing. You will implement tests with Playwright (TypeScript) or Python, manage evaluation datasets, define ground truth, and integrate tests into CI/CD pipelines, ensuring robust

Qualifications

• 5+ years of QA / SDET / Test Automation experience.
• 1-2+ years hands-on testing of AI/LLM apps.
• Hands-on experience with LLM evaluation / AI evaluation.
• Experience with evaluation datasets and ground truth.
• Understanding of LLM judges / AI-as-a-judge and scoring.
• Experience testing nondeterministic AI outputs.
• LangSmith experience.
• LangGraph/agent workflow experience.
• Strong Playwright experience, preferably with TypeScript.
• Strong Python and/or TypeScript skills.
• API, integration, regression, and end-to-end testing.
• Experience CI/CD integration.

Responsibilities

• Design and implement automated tests for AI/LLM-powered apps using Playwright and TS/Python.
• Create and maintain LLM evaluation datasets and regression suites.
• Define ground truth, scoring criteria, and pass thresholds for AI outputs.
• Evaluate nondeterministic LLM results with evaluators and comparison methods.
• Validate accuracy, relevance, groundedness, hallucinations, and consistency of AI responses.
• Test AI workflows including prompts, tools, and multi-step flows.
• Trace/test AI behavior with LangSmith and similar frameworks.
• Investigate failures to determine model vs data vs product defects.
• Build automated regression checks and CI/CD integration.
• Improve test reliability and fix flaky tests.

Skills

Playwright

TypeScript

Python

LangSmith

LangGraph

AI/LLM testing

SDET

CI/CD

Git

Agile

Tools

Playwright

LangSmith

LangGraph

GitHub Copilot

Jenkins/GitHub Actions

Job description

Brilliant Infotech Inc. in Birmingham, AL seeks a Senior Quality Engineer to lead AI/LLM test automation. You will build automated testing and evaluation frameworks for AI-powered apps, focusing on accuracy, relevance, and reliability of responses.

The role emphasizes nondeterministic output evaluation and regression testing. You will implement tests with Playwright (TypeScript) or Python, manage evaluation datasets, define ground truth, and integrate tests into CI/CD pipelines, ensuring robust