Senior ML & Evaluation Engineer - Remote + Flexible Hours

EPAM Systems; Portugal; Remote; EUR 90,000 - 140,000; Full time

Benefits offered by this job

Competitive compensation

Flexible work hours

Career growth opportunities

Job summary

EPAM Systems in Portugal is seeking a Senior ML / Evaluation Engineer to own the design and implementation of advanced evaluation frameworks for a cloud-native AI agent platform. You will craft evaluation strategies, deploy CI/CD gates, and integrate LLM-as-judge and deterministic evaluators into automated pipelines.

You will work with AWS AgentCore modules, Lambda-based evaluators, and observability signals via OpenTelemetry and CloudWatch to monitor quality across production.

Qualifications

• 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development.
• Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders).
• Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance.
• Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators).
• Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring.

Responsibilities

• Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications.
• Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks.
• Develop deterministic evaluators as AWS Lambda functions for rule-based validation.
• Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds.
• Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines.
• Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails.
• Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment.
• Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms.
• Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability.

Skills

ML engineering

AI evaluation frameworks

Python

CI/CD pipelines

LLM evaluation

AWS Lambda

OpenTelemetry

CloudWatch

Tools

AWS

CI/CD tooling

Job description

EPAM Systems in Portugal is seeking a Senior ML / Evaluation Engineer to own the design and implementation of advanced evaluation frameworks for a cloud-native AI agent platform. You will craft evaluation strategies, deploy CI/CD gates, and integrate LLM-as-judge and deterministic evaluators into automated pipelines.

You will work with AWS AgentCore modules, Lambda-based evaluators, and observability signals via OpenTelemetry and CloudWatch to monitor quality across production.