Senior ML Evaluation Architect for Enterprise AI

Intellias; Poland; On-site; PLN 180,000 - 240,000; Full time

Job summary

Intellias is seeking a Senior ML / Evaluation Engineer to define and implement quality standards for enterprise-grade AI agents and LLM-powered applications. You will design evaluation frameworks, build custom evaluation pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will collaborate with AI Platform Engineers, ML Engineers, and DevOps teams to ensure reliable, measurable, and production-ready AI systems through scalable evaluation, observability, and

Qualifications

• 5+ years ML engineering or AI platform engineering
• Experience designing evaluation frameworks for LLMs and AI agents
• Implementation of deterministic quality checks via custom evaluators

Responsibilities

• Design, implement, and maintain enterprise-grade evaluation frameworks for LLMs, AI agents, and multi-step AI workflows.
• Develop and optimize LLM-as-a-judge evaluators to assess helpfulness, correctness, and policy compliance.
• Build custom Python-based evaluators using AWS Lambda for deterministic validation and workflow quality checks.
• Define evaluation standards, scoring methodologies, and pass/fail criteria across AI platforms.
• Integrate evaluation workflows into CI/CD pipelines and automated deployment quality gates.

Skills

AWS AgentCore Evaluation

Custom Python evaluators

Evaluation scopes (TRACE/TOOL_CALL/SEM

OTel spans from AgentCore Observabilit

Tools

AWS Lambda

Python

OpenTelemetry

Job description

Intellias is seeking a Senior ML / Evaluation Engineer to define and implement quality standards for enterprise-grade AI agents and LLM-powered applications. You will design evaluation frameworks, build custom evaluation pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will collaborate with AI Platform Engineers, ML Engineers, and DevOps teams to ensure reliable, measurable, and production-ready AI systems through scalable evaluation, observability, and