Senior ML Evaluation Engineer — Remote (Spain)

Intellias; Alicante; Presencial; EUR 70.000 - 100.000; Jornada completa

Descripción de la vacante

Intellias is seeking a Senior ML / Evaluation Engineer to define and implement enterprise-grade evaluation standards for AI agents and LLM-powered applications. You will design evaluation frameworks, build custom pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will collaborate with AI Platform Engineers, ML Engineers, and DevOps teams to ensure reliable, production-ready AI systems with scalable evaluation and observability practices.

Formación

• 5+ years of ML engineering or AI platform engineering.
• LLM evaluation framework design and implementation.
• Custom evaluator implementation for deterministic quality checks.
• CI/CD deployment gate design for ML model or agent quality.
• AWS AgentCore Evaluation capabilities for on-demand evaluations.

Responsabilidades

• Design, implement, and maintain enterprise-grade evaluation frameworks for LLMs, AI agents, and multi-step AI workflows.
• Develop LLM-as-a-judge evaluators to assess helpfulness, correctness, and policy compliance.
• Build Python-based evaluators using AWS Lambda for deterministic validation and workflow checks.
• Define evaluation standards, dimensions, and pass/fail criteria across AI platforms.
• Integrate evaluation workflows into CI/CD pipelines and establish automated deployment quality gates.

Conocimientos

ML engineering

LLM evaluation framework design

Custom evaluator implementation

CI/CD for ML

AWS AgentCore

LLM-as-judge evaluator

Python Lambda evaluators

Evaluation levels (TRACE, TOOL_CALL, S

OpenTelemetry

Enterprise evaluation standards

Herramientas

AWS Lambda

AgentCore Evaluation

CloudWatch

OpenTelemetry

Descripción del empleo

Intellias is seeking a Senior ML / Evaluation Engineer to define and implement enterprise-grade evaluation standards for AI agents and LLM-powered applications. You will design evaluation frameworks, build custom pipelines, and establish automated quality gates across the AI delivery lifecycle.

You will collaborate with AI Platform Engineers, ML Engineers, and DevOps teams to ensure reliable, production-ready AI systems with scalable evaluation and observability practices.