GenAI/LLM Quality Engineer — Testing & Validation

BrickRed Systems LLC; Bellevue (WA); On-site; USD 120,000 - 160,000; Full time

Job summary

BrickRed Systems LLC is seeking a GenAI/LLM Test Engineer to join our expanding AI product testing team in Bellevue, WA. You will design and execute test plans for GenAI features, prompts, APIs, and data pipelines, and build automated functional, regression, integration, and API tests.

You will evaluate model outputs for accuracy, bias, hallucination, and robustness, and collaborate with Data Scientists, ML Engineers, and DevOps to define quality metrics and CI/CD integration.

Qualifications

• Experience in GenAI/LLM product testing and AI evaluation.
• Proficient in Python and SQL.
• Familiar with AI evaluation frameworks.
• Experience with test automation, regression and API testing.
• Ability to perform performance and adversarial testing.

Responsibilities

• Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications.
• Build automated functional, regression, integration, and API test suites.
• Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness.
• Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites.
• Perform adversarial/red-team, performance, scalability, and reliability testing.
• Validate AI data pipelines for data quality, schema, and drift.
• Integrate automated and AI evaluation testing into CI/CD pipelines.
• Collaborate with Data Scientists, ML Engineers, and DevOps teams to define quality metrics and acceptance criteria.

Skills

LLM/GenAI Product Testing

Python & SQL

AI Evaluation Frameworks

Test Automation

Performance Testing

Adversarial Testing

Generative AI, LLMs & RAG

Tools

CI/CD Automation

Job description

BrickRed Systems LLC is seeking a GenAI/LLM Test Engineer to join our expanding AI product testing team in Bellevue, WA. You will design and execute test plans for GenAI features, prompts, APIs, and data pipelines, and build automated functional, regression, integration, and API tests.

You will evaluate model outputs for accuracy, bias, hallucination, and robustness, and collaborate with Data Scientists, ML Engineers, and DevOps to define quality metrics and CI/CD integration.