Senior ML / Evaluation Engineer

EPAM Systems; Portugal; Remote; EUR 90,000 - 140,000; Full time

Benefits offered by this job

Competitive compensation

Flexible work hours

Career growth opportunities

Job summary

EPAM Systems in Portugal is seeking a Senior ML / Evaluation Engineer to own the design and implementation of advanced evaluation frameworks for a cloud-native AI agent platform. You will craft evaluation strategies, deploy CI/CD gates, and integrate LLM-as-judge and deterministic evaluators into automated pipelines.

You will work with AWS AgentCore modules, Lambda-based evaluators, and observability signals via OpenTelemetry and CloudWatch to monitor quality across production.

Qualifications

• 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development.
• Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders).
• Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance.
• Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators).
• Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring.

Responsibilities

• Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications.
• Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks.
• Develop deterministic evaluators as AWS Lambda functions for rule-based validation.
• Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds.
• Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines.
• Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails.
• Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment.
• Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms.
• Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability.

Skills

ML engineering

AI evaluation frameworks

Python

CI/CD pipelines

LLM evaluation

AWS Lambda

OpenTelemetry

CloudWatch

Tools

AWS

CI/CD tooling

Job description

We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create evaluation strategies that combine LLM-as-judge grading with deterministic checks, define enterprise evaluation standards, and implement CI/CD deployment gates to enforce quality metrics prior to release. This position requires strong expertise in ML system testing, evaluation design, and integration into automated pipelines for agentic environments.

Responsibilities

• Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications
• Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks
• Develop deterministic evaluators as AWS Lambda functions for rule-based validation
• Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds
• Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines
• Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails
• Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment
• Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms
• Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability

Requirements

• 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development
• Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders)
• Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance
• Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators)
• Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring

Nice to have

• Familiarity with AWS AgentCore Evaluations API (CreateEvaluation, GetEvaluationResult)
• Exposure to AWS Bedrock Guardrails for compliance and sensitive data validation
• Experience integrating evaluation metrics into AWS CloudWatch for monitoring and alerting
• Knowledge of OTel instrumentation and trace ingestion for quality scoring inputs

We offer

• Competitive compensation depending on experience and skills
• Variety of projects within one company
• Being a part of a project following engineering excellence standards
• Individual career path and professional growth opportunities
• Internal events and communities
• Flexible work hours