Senior Python AI Evaluation Engineer — Production Quality

EPAM Systems; Turkey; On-site; TRY 4,442,000 - 6,417,000; Full time

Benefits offered by this job

Continuous upskilling

Career development

Private health insurance

English courses

Mentoring programs

Learning platforms

Job summary

EPAM Systems is seeking a Python AI Evaluation Engineer to join the Enterprise Agent Development Platform. You will own the capabilities that measure and validate AI solutions, designing evaluation approaches, developing custom evaluators, and embedding quality checks into the software delivery lifecycle.

This role requires 5+ years in ML/AI engineering, strong Python skills, and hands-on experience with LLM/GenAI evaluation.

Qualifications

• 5+ years of experience in ML/AI engineering or AI Platform Engineering.
• Strong Python development experience.
• Hands-on experience with LLM/GenAI evaluation.
• Experience designing and implementing evaluation frameworks.
• Experience developing custom or deterministic evaluators.
• Experience integrating AI/ML quality checks into CI/CD.
• Good understanding of LLM and AI agent architectures.

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents
• Develop LLM-as-a-Judge and deterministic, code-based evaluators
• Build custom Python evaluators for quality and behavioral checks
• Define evaluation criteria, metrics, thresholds, and acceptance rules
• Evaluate agent behavior across individual responses, tool calls, and complete workflows
• Work with OpenTelemetry traces and spans as evaluation data
• Integrate evaluations into CI/CD pipelines and automated deployment gates
• Enable continuous quality monitoring of solutions in production
• Establish reusable evaluation patterns and engineering standards
• Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process

Skills

Python development

ML/AI engineering

LLM/GenAI evaluation

CI/CD integration

OpenTelemetry data

Tools

AWS

Bedrock Guardrails

LangGraph

Strands Agents

Job description

EPAM Systems is seeking a Python AI Evaluation Engineer to join the Enterprise Agent Development Platform. You will own the capabilities that measure and validate AI solutions, designing evaluation approaches, developing custom evaluators, and embedding quality checks into the software delivery lifecycle.

This role requires 5+ years in ML/AI engineering, strong Python skills, and hands-on experience with LLM/GenAI evaluation.