Remote Senior ML Evaluation Engineer – AI Quality

Intellias; Spain (TX); On-site; EUR 70,000 - 100,000; Full time

Job summary

Intellias is seeking a Senior ML / Evaluation Engineer to help define and implement quality standards for enterprise-grade AI agents and LLM-powered applications. You will design evaluation frameworks, build custom evaluation pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will work with AI Platform Engineers, ML Engineers, and DevOps teams to ensure production-ready AI systems.

Qualifications

• 5+ years ML engineering or AI platform engineering.
• LLM evaluation framework design and implementation.
• Custom evaluator implementation for deterministic quality checks.
• AWS AgentCore Evaluation (on-demand mode for CI/CD gates, online mode for production sampling).
• Custom code-based Lambda evaluators (Python - deterministic checks).
• Evaluation levels (TRACE, TOOL_CALL, SESSION).
• OTel spans from AWS AgentCore Observability as evaluation input.

Responsibilities

• Design, implement, and maintain enterprise-grade evaluation frameworks for LLMs and AI agents.
• Build and optimize LLM-as-a-judge evaluators for multiple dimensions (helpfulness, correctness, consistency, policy).
• Develop Python-based evaluators using AWS Lambda for deterministic validation and workflow checks.
• Define evaluation standards, scoring methodologies, and pass/fail criteria across AI platforms.
• Design evaluation strategies at TRACE, TOOL_CALL, and SESSION levels.
• Integrate evaluation workflows into CI/CD pipelines with automated quality gates.
• Leverage AWS AgentCore and OpenTelemetry signals as evaluation inputs for quality analysis.
• Collaborate with platform, security, and AI engineering teams to improve reliability.

Skills

ML engineering

AI platform engineering

LLM evaluation

Custom evaluators

AWS AgentCore

Python evaluators

Evaluation levels

Observability data

Tools

Python

AWS Lambda

OpenTelemetry

AgentCore

CI/CD tooling

Job description

Intellias is seeking a Senior ML / Evaluation Engineer to help define and implement quality standards for enterprise-grade AI agents and LLM-powered applications. You will design evaluation frameworks, build custom evaluation pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will work with AI Platform Engineers, ML Engineers, and DevOps teams to ensure production-ready AI systems.