Senior AI Evaluation Platform Lead

EPAM Systems; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

EPAM Systems is seeking an experienced AI Evaluation Lead to architect and oversee evaluation infrastructure for non-deterministic AI systems. You will design evaluation strategies, build shared platforms, and ensure trustworthy grading across retrieval, grounding and tool-use.

You will lead the development of Arabic evaluation capabilities, implement online evaluation, and integrate quality gates into CI/CD. Strong Python and CC/Java scripting, plus leadership experience, are required.

Qualifications

• 5+ years building evaluation/test infrastructure and frameworks.

Responsibilities

• Set evaluation strategy: what gets measured, at which layer, with what methodology, and how results feed product decisions.
• Build the shared platform with evaluation harnesses, dataset versioning, automated grading, and dashboards for leadership.
• Make grading trustworthy through judge model selection, rubric design, calibration against human labels, and knowing when automated grading cannot be trusted.
• Evaluate retrieval and agents for grounding and citation correctness, tool-use validation, multi-step reasoning and failure recovery.
• Develop Arabic golden sets and judges calibrated for Arabic rather than transfer from English.
• Run online evaluation and drift detection.
• Conduct prompt injection, jailbreak and data-leakage red-teaming.
• Wire evaluation and test automation into CI/CD as quality gates.

Skills

Python

TypeScript/Java

CI/CD

Statistics

English B2+

Leadership

Non-deterministic evaluation

Tools

LangSmith

Langfuse

Promptfoo

Braintrust

DeepEval

Job description

EPAM Systems is seeking an experienced AI Evaluation Lead to architect and oversee evaluation infrastructure for non-deterministic AI systems. You will design evaluation strategies, build shared platforms, and ensure trustworthy grading across retrieval, grounding and tool-use.

You will lead the development of Arabic evaluation capabilities, implement online evaluation, and integrate quality gates into CI/CD. Strong Python and CC/Java scripting, plus leadership experience, are required.