AI Evaluation Engineer (Fully Remote)

Undisclosed; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Undisclosed is seeking a remote, global team member to design and build evaluation infrastructure for production-grade AI agents. The role owns the architecture from the ground up and collaborates across time zones.

You will implement CI/CD pipelines, golden datasets, adversarial testing, and advanced statistical evaluation, observability, and structured output validation to ensure reliability.

Qualifications

• Strong Python/TypeScript and LLM evaluation experience.
• Experience with CI/CD integration and owning architecture.

Responsibilities

• Design and build evaluation infrastructure for production-grade AI agents.
• Build CI/CD pipelines, golden datasets, and adversarial testing for AI reliability.
• Implement statistical evaluation, observability, and structured output validation.

Skills

Python

TypeScript

LLM evaluation

Tools

CI/CD tools

Observability tooling

Job description

• Design and build evaluation infrastructure for production-grade AI agents.
• Own the architecture from the ground up in a remote, global team.

Key Responsibilities:

• Build CI/CD pipelines, golden datasets, and adversarial testing for AI reliability.
• Implement statistical evaluation, observability, and structured output validation.

Requirements:

• Strong Python/TypeScript and LLM evaluation experience.
• CI/CD integration and independent architecture ownership skills.