Senior ML / Evaluation Engineer

EPAM Systems, Inc.; Turkey; Hybrid; TRY 4,420,000 - 7,367,000; Full time

Benefits offered by this job

Continuous upskilling

Mentoring programs

Certification support

Learning platforms access

Extra leave days

Referral bonuses

Private health insurance

Well-being events

Recreation areas

Snacks and beverages

IT equipment (HUP)

Job summary

EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.

You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.

Qualifications

• 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering.
• Strong Python development experience.
• Hands-on experience with LLM/GenAI evaluation.
• Experience designing and implementing evaluation frameworks.
• Experience developing custom or deterministic evaluators.
• Experience integrating AI/ML quality checks into CI/CD.

Responsibilities

• Design and implement evaluation frameworks for LLM-based applications and AI agents.
• Develop LLM-as-a-Judge and deterministic, code-based evaluators.
• Build custom Python evaluators for quality and behavioral checks.
• Define evaluation criteria, metrics, thresholds, and acceptance rules.
• Evaluate agent behavior across responses, tool calls, and complete workflows.
• Work with OpenTelemetry traces and spans as evaluation data.
• Integrate evaluations into CI/CD pipelines and automated deployment gates.
• Enable continuous quality monitoring of solutions in production.

Skills

Python

LLM evaluation

CI/CD

Deterministic evaluators

Quality checks

AI/ML engineering

Tools

OpenTelemetry

LangGraph

Strands Agents

AWS Bedrock Guardrails

CloudWatch

Job description

Join the Enterprise Agent Development Platform project at EPAM. We are building a cloud-native platform that enables engineering teams to develop, deploy, and operate AI agents in production.The platform combines modern agent frameworks, AWS infrastructure, CI/CD, observability, and governance to make AI development faster and more reliable.As a Python AI Evaluation Engineer, you will own a key part of this platform: building the capabilities that measure and validate the quality of AI solutions. You will design evaluation approaches, develop custom evaluators, and integrate quality checks into the software delivery lifecycle.This role is a strong fit for an engineer who enjoys solving new GenAI challenges and turning them into practical, automated engineering solutions.ResponsibilitiesDesign and implement evaluation frameworks for LLM-based applications and AI agentsDevelop LLM-as-a-Judge and deterministic, code-based evaluatorsBuild custom Python evaluators for quality and behavioral checksDefine evaluation criteria, metrics, thresholds, and acceptance rulesEvaluate agent behavior across individual responses, tool calls, and complete workflowsWork with OpenTelemetry traces and spans as evaluation dataIntegrate evaluations into CI/CD pipelines and automated deployment gatesEnable continuous quality monitoring of solutions in productionEstablish reusable evaluation patterns and engineering standardsWork closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development processRequirements5+ years of experience in ML Engineering, AI Engineering, or AI Platform EngineeringStrong Python development experienceHands-on experience with LLM/GenAI evaluationExperience designing and implementing evaluation frameworksExperience developing custom or deterministic evaluatorsExperience integrating AI/ML quality checks into CI/CDGood understanding of LLM and AI agent architecturesNice to haveHands-on experience with AWS AgentCore EvaluationExperience with AWS Bedrock Guardrails, including PII detectionKnowledge of CloudWatch metrics and production monitoringExperience with OpenTelemetryFamiliarity with LangGraph, Strands Agents, or similar agent frameworksWe offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.