ML Evaluation Engineer - LLM Quality & CI/CD Automation

EPAM Systems, Inc.; Turkey; Hybrid; TRY 4,420,000 - 7,367,000; Full time

Benefits offered by this job

Continuous upskilling

Mentoring programs

Certification support

Learning platforms access

Extra leave days

Referral bonuses

Private health insurance

Well-being events

Recreation areas

Snacks and beverages

IT equipment (HUP)

Job summary

EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.

You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.

Qualifications

• 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering.
• Strong Python development experience.
• Hands-on experience with LLM/GenAI evaluation.
• Experience designing and implementing evaluation frameworks.
• Experience developing custom or deterministic evaluators.
• Experience integrating AI/ML quality checks into CI/CD.

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents.
• Develop LLM-as-a-Judge and deterministic, code-based evaluators.
• Build custom Python evaluators for quality and behavioral checks.
• Define evaluation criteria, metrics, thresholds, and acceptance rules.
• Evaluate agent behavior across responses, tool calls, and complete workflows.
• Work with OpenTelemetry traces and spans as evaluation data.
• Integrate evaluations into CI/CD pipelines and automated deployment gates.
• Enable continuous quality monitoring of solutions in production.

Skills

Python

LLM evaluation

CI/CD

Deterministic evaluators

Quality checks

AI/ML engineering

Tools

OpenTelemetry

LangGraph

Strands Agents

AWS Bedrock Guardrails

CloudWatch

Job description

EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.

You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.