AI Solution/Principal Engineer

EPAM Systems; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

EPAM Systems is seeking an experienced AI Evaluation Lead to architect and oversee evaluation infrastructure for non-deterministic AI systems. You will design evaluation strategies, build shared platforms, and ensure trustworthy grading across retrieval, grounding and tool-use.

You will lead the development of Arabic evaluation capabilities, implement online evaluation, and integrate quality gates into CI/CD. Strong Python and CC/Java scripting, plus leadership experience, are required.

Qualifications

• 5+ years building evaluation/test infrastructure and frameworks.

Responsibilities

• Set evaluation strategy: what gets measured, at which layer, with what methodology, and how results feed product decisions.
• Build the shared platform with evaluation harnesses, dataset versioning, automated grading, and dashboards for leadership.
• Make grading trustworthy through judge model selection, rubric design, calibration against human labels, and knowing when automated grading cannot be trusted.
• Evaluate retrieval and agents for grounding and citation correctness, tool-use validation, multi-step reasoning and failure recovery.
• Develop Arabic golden sets and judges calibrated for Arabic rather than transfer from English.
• Run online evaluation and drift detection.
• Conduct prompt injection, jailbreak and data-leakage red-teaming.
• Wire evaluation and test automation into CI/CD as quality gates.

Skills

Python

TypeScript/Java

CI/CD

Statistics

English B2+

Leadership

Non-deterministic evaluation

Tools

LangSmith

Langfuse

Promptfoo

Braintrust

DeepEval

Job description

Responsibilities

• Set evaluation strategy: what gets measured, at which layer, with what methodology, and how results feed product decisions
• Build the shared platform, including evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection, and reporting leadership can read
• Make grading trustworthy through judge model selection, rubric design, calibration against human labels, and knowing when automated grading cannot be trusted
• Evaluate retrieval and agents for grounding and citation correctness, tool-use validation, multi-step reasoning and failure recovery
• Develop Arabic golden sets and judges calibrated for Arabic rather than assumed to transfer from English
• Run online evaluation and drift detection
• Conduct prompt injection, jailbreak and data-leakage red-teaming
• Wire evaluation and test automation together as quality gates in CI/CD

Requirements

• 5+ years of experience building evaluation or test infrastructure others depend on, including harnesses, shared libraries and frameworks
• At least 1 year of relevant leadership experience
• Expertise in AI evaluation covering output quality, retrieval and grounding, and regression detection on non-deterministic behavior
• Proficiency in LLM-as-judge methodology: rubric design, calibration against human labels, and understanding its failure modes
• Advanced proficiency in Python, at the depth needed to build shared libraries, with intermediate competency in TypeScript or Java
• Knowledge of statistics for non-deterministic systems, including sampling, confidence intervals, inter-rater agreement and significance
• Skills in CI/CD framework design across API, web and data surfaces, with test selection, parallelization and flake management
• Background in golden sets and dataset versioning
• English proficiency at an Upper-Intermediate level (B2) or higher
• Nice to have Familiarity with ragas, DeepEval, Promptfoo, Braintrust, LangSmith or Langfuse
• Expertise in Arabic evaluation, including golden sets, dialect coverage, right-to-left validation and judge calibration
• Background in adversarial and security testing, including red-teaming
• Capability to perform voice and conversational evaluation, and performance testing against AI services
• Experience delivering in a regulated or government environment