Senior ML / Evaluation Engineer

EPAM Systems; Turkey; On-site; TRY 4,442,000 - 6,417,000; Full time

Benefits offered by this job

Continuous upskilling

Career development

Private health insurance

English courses

Mentoring programs

Learning platforms

Job summary

EPAM Systems is seeking a Python AI Evaluation Engineer to join the Enterprise Agent Development Platform. You will own the capabilities that measure and validate AI solutions, designing evaluation approaches, developing custom evaluators, and embedding quality checks into the software delivery lifecycle.

This role requires 5+ years in ML/AI engineering, strong Python skills, and hands-on experience with LLM/GenAI evaluation.

Qualifications

• 5+ years of experience in ML/AI engineering or AI Platform Engineering.
• Strong Python development experience.
• Hands-on experience with LLM/GenAI evaluation.
• Experience designing and implementing evaluation frameworks.
• Experience developing custom or deterministic evaluators.
• Experience integrating AI/ML quality checks into CI/CD.
• Good understanding of LLM and AI agent architectures.

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents
• Develop LLM-as-a-Judge and deterministic, code-based evaluators
• Build custom Python evaluators for quality and behavioral checks
• Define evaluation criteria, metrics, thresholds, and acceptance rules
• Evaluate agent behavior across individual responses, tool calls, and complete workflows
• Work with OpenTelemetry traces and spans as evaluation data
• Integrate evaluations into CI/CD pipelines and automated deployment gates
• Enable continuous quality monitoring of solutions in production
• Establish reusable evaluation patterns and engineering standards
• Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process

Skills

Python development

ML/AI engineering

LLM/GenAI evaluation

CI/CD integration

OpenTelemetry data

Tools

AWS

Bedrock Guardrails

LangGraph

Strands Agents

Job description

Join the Enterprise Agent Development Platform project at EPAM. We are building a cloud-native platform that enables engineering teams to develop, deploy, and operate AI agents in production.

The platform combines modern agent frameworks, AWS infrastructure, CI/CD, observability, and governance to make AI development faster and more reliable.

As a Python AI Evaluation Engineer, you will own a key part of this platform: building the capabilities that measure and validate the quality of AI solutions. You will design evaluation approaches, develop custom evaluators, and integrate quality checks into the software delivery lifecycle.

This role is a strong fit for an engineer who enjoys solving new GenAI challenges and turning them into practical, automated engineering solutions.

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents
• Develop LLM-as-a-Judge and deterministic, code-based evaluators
• Build custom Python evaluators for quality and behavioral checks
• Define evaluation criteria, metrics, thresholds, and acceptance rules
• Evaluate agent behavior across individual responses, tool calls, and complete workflows
• Work with OpenTelemetry traces and spans as evaluation data
• Integrate evaluations into CI/CD pipelines and automated deployment gates
• Enable continuous quality monitoring of solutions in production
• Establish reusable evaluation patterns and engineering standards
• Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process

Requirements

• 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering
• Strong Python development experience
• Hands‑on experience with LLM/GenAI evaluation
• Experience designing and implementing evaluation frameworks
• Experience developing custom or deterministic evaluators
• Experience integrating AI/ML quality checks into CI/CD
• Good understanding of LLM and AI agent architectures

Nice to have

• Hands‑on experience with AWS AgentCore Evaluation
• Experience with AWS Bedrock Guardrails, including PII detection
• Knowledge of CloudWatch metrics and production monitoring
• Experience with OpenTelemetry
• Familiarity with LangGraph, Strands Agents, or similar agent frameworks

We offer

• CONTINUOUS UPSKILLING, LEARNING & DEVELOPMENT
  • Diversity of tasks and projects
  • Assessment center for objective review of competency level
  • Personal development plan
  • Mentoring programs and leadership development
  • Certification and professional development support
  • Access to learning platforms including more than 2,500 internal courses
  • English courses taught by certified teachers
• CORPORATE BENEFITS
  • Extra leave days
  • Referral bonuses
• COMPENSATION PACKAGE
  • Competitive compensation paid in USD
  • Regular salary and performance reviews
• MEDICAL & HEALTHCARE
  • Private health insurance
  • Well‑being events
• WORKING ENVIRONMENT
  • Recreation areas and kitchens
  • Tea, coffee and snacks
  • Sports equipment and game consoles
  • IT Equipment
  • Microsoft’s Software Assurance Home Use Program (HUP)

Please note that our Talent Attraction Team reviews applications and CVs submitted in English.

EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI‑Native enterprises, driving measurable value from innovation and digital investments.