GenAI/LLM Test Engineer

BrickRed Systems LLC; Bellevue (WA); On-site; USD 120,000 - 160,000; Full time

Job summary

BrickRed Systems LLC is seeking a GenAI/LLM Test Engineer to join our expanding AI product testing team in Bellevue, WA. You will design and execute test plans for GenAI features, prompts, APIs, and data pipelines, and build automated functional, regression, integration, and API tests.

You will evaluate model outputs for accuracy, bias, hallucination, and robustness, and collaborate with Data Scientists, ML Engineers, and DevOps to define quality metrics and CI/CD integration.

Qualifications

• Experience in GenAI/LLM product testing and AI evaluation.
• Proficient in Python and SQL.
• Familiar with AI evaluation frameworks.
• Experience with test automation, regression and API testing.
• Ability to perform performance and adversarial testing.

Responsibilities

• Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications.
• Build automated functional, regression, integration, and API test suites.
• Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness.
• Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites.
• Perform adversarial/red-team, performance, scalability, and reliability testing.
• Validate AI data pipelines for data quality, schema, and drift.
• Integrate automated and AI evaluation testing into CI/CD pipelines.
• Collaborate with Data Scientists, ML Engineers, and DevOps teams to define quality metrics and acceptance criteria.

Skills

LLM/GenAI Product Testing

Python & SQL

AI Evaluation Frameworks

Test Automation

Performance Testing

Adversarial Testing

Generative AI, LLMs & RAG

Tools

CI/CD Automation

Job description

We are looking for a GenAI/LLM Test Engineer with strong experience in AI/ML product testing, Python, AI evaluation frameworks, SQL, and test automation. The role will focus on testing LLM-based applications, RAG systems, AI agents, prompts, model outputs, and data pipelines.

Required Skills:

• LLM/GenAI Product Testing
• Python & SQL
• AI Evaluation Frameworks – Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, TruLens
• Test Automation, Regression & API Testing
• Performance & Adversarial Testing
• Generative AI, LLMs & RAG

Nice to Have:

• CI/CD Automation

Key Responsibilities:

• Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications.
• Build automated functional, regression, integration, and API test suites.
• Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness.
• Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites.
• Perform adversarial/red-team, performance, scalability, and reliability testing.
• Validate AI data pipelines for data quality, schema, and drift.
• Integrate automated and AI evaluation testing into CI/CD pipelines.
• Collaborate with Data Scientists, ML Engineers, and DevOps teams to define quality metrics and acceptance criteria.

Environment: Primarily Microsoft Azure and Snowflake, with GenAI initiatives leveraging OpenAI and Claude models.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions.

BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers. Through its focus on delivering specialized talent and technology expertise, BrickRed Systems supports organizations in building scalable teams and executing critical business initiatives across diverse industries and functional areas.