ML Evaluation Engineer - LLM Quality & CI/CD Automation

EPAM Systems

Turkey

Hybrid

TRY 4,420,000 - 7,367,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Continuous upskilling
Mentoring programs
Certification support
Learning platforms access
Extra leave days
Referral bonuses
Private health insurance
Well-being events
Recreation areas
Snacks and beverages
IT equipment (HUP)

Job summary

EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.

You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.

Qualifications

  • 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering.
  • Strong Python development experience.
  • Hands-on experience with LLM/GenAI evaluation.
  • Experience designing and implementing evaluation frameworks.
  • Experience developing custom or deterministic evaluators.
  • Experience integrating AI/ML quality checks into CI/CD.

Responsibilities

  • Design and implement evaluation frameworks for LLM-based applications and AI agents.
  • Develop LLM-as-a-Judge and deterministic, code-based evaluators.
  • Build custom Python evaluators for quality and behavioral checks.
  • Define evaluation criteria, metrics, thresholds, and acceptance rules.
  • Evaluate agent behavior across responses, tool calls, and complete workflows.
  • Work with OpenTelemetry traces and spans as evaluation data.
  • Integrate evaluations into CI/CD pipelines and automated deployment gates.
  • Enable continuous quality monitoring of solutions in production.

Skills

Python
LLM evaluation
CI/CD
Deterministic evaluators
Quality checks
AI/ML engineering

Tools

OpenTelemetry
LangGraph
Strands Agents
AWS Bedrock Guardrails
CloudWatch

Job description

EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.

You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML / Evaluation Engineer
Senior ML / Evaluation Engineer

EPAM Systems • Turkey

Hybrid
TRY 4,420,000 - 7,367,000
Continuous upskilling
Mentoring programs
Certification support
+8
Senior Python AI Evaluation Engineer — Production Quality
Senior Python AI Evaluation Engineer — Production Quality

EPAM Systems Inc • Turkey

On-site
TRY 4,442,000 - 6,417,000
Continuous upskilling
Career development
Private health insurance
+3
Senior ML / Evaluation Engineer
Senior ML / Evaluation Engineer

EPAM Systems Inc • Turkey

On-site
TRY 4,442,000 - 6,417,000
Continuous upskilling
Career development
Private health insurance
+3
Senior AI Engineer
Senior AI Engineer

Kogniser • Fatih

On-site
TRY 900,000 - 1,300,000
Senior AI Solutions Architect
Senior AI Solutions Architect

EPAM Systems • Turkey

On-site
TRY 5,914,000 - 8,871,000
Private health insurance
Learning & development programs
English courses
+2
Forward Deployed Engineer/Chief Role
Forward Deployed Engineer/Chief Role

EPAM Systems Inc • Turkey

On-site
TRY 4,423,000 - 7,371,000
Extra leave days
Referral bonuses
Private health insurance
+5
Senior AI Engineer/ Node.js, Python, Typescript
Senior AI Engineer/ Node.js, Python, Typescript

EPAM Systems Inc • Turkey

On-site
TRY 3,455,000 - 5,429,000
Private health insurance
Mentoring programs
Professional development support
+1
AI Engineer: Build Scalable LLM Apps & Pipelines
AI Engineer: Build Scalable LLM Apps & Pipelines

Coderspace • Turkey

On-site
TRY 550,000 - 900,000
Meal allowance
Private health insurance
Commuting support
+8
Senior AI Solution Engineer
Senior AI Solution Engineer

EPAM Systems • Turkey

On-site
TRY 5,914,000 - 8,871,000
Private health insurance
Learning & development programs
English courses
+2
Lead Python ML Engineer - NLP, DL, Security
Lead Python ML Engineer - NLP, DL, Security

EPAM Systems Inc • Turkey

On-site
TRY 5,831,000 - 7,775,000
Continuous upskilling
Private health insurance
English courses
+3