AI Evaluation Engineer: Production-Grade Testing CI/CD

Undisclosed; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Undisclosed is seeking a remote, global team member to design and build evaluation infrastructure for production-grade AI agents. The role owns the architecture from the ground up and collaborates across time zones.

You will implement CI/CD pipelines, golden datasets, adversarial testing, and advanced statistical evaluation, observability, and structured output validation to ensure reliability.

Qualifications

• Strong Python/TypeScript and LLM evaluation experience.
• Experience with CI/CD integration and owning architecture.

Responsibilities

• Design and build evaluation infrastructure for production-grade AI agents.
• Build CI/CD pipelines, golden datasets, and adversarial testing for AI reliability.
• Implement statistical evaluation, observability, and structured output validation.

Skills

Python

TypeScript

LLM evaluation

Tools

CI/CD tools

Observability tooling

Job description

Undisclosed is seeking a remote, global team member to design and build evaluation infrastructure for production-grade AI agents. The role owns the architecture from the ground up and collaborates across time zones.

You will implement CI/CD pipelines, golden datasets, adversarial testing, and advanced statistical evaluation, observability, and structured output validation to ensure reliability.