AI Solution Architect & Platform Lead

EPAM Systems; Turkey; On-site; TRY 5,938,000 - 9,401,000; Full time

Benefits offered by this job

Private health insurance

Career development

Mentoring programs

Learning platform access

English courses

Job summary

EPAM Systems in Turkey seeks an AI Solution/Principal Engineer to lead evaluation of a sovereign, multi-tenant AI platform and its applied products operating in Arabic and English. You will own how we measure and improve non-deterministic AI across assistants, retrieval, tools, and infrastructure, shaping products with trustworthy, evidence-based results.

Responsibilities include building evaluation harnesses, dataset versioning, online drift detection, and CI/CD integrations to gate quality

Qualifications

• 5+ years building evaluation or test infrastructure and shared libraries.
• Leadership experience of at least 1 year.
• Expertise in AI evaluation spanning output quality, retrieval and grounding.
• Proficiency in Python; intermediate in TypeScript.
• Statistics knowledge for non-deterministic systems.

Responsibilities

• Define evaluation strategy: what gets measured, layers, methodology, and feed product decisions.
• Build evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection and reporting.
• Ensure grading trustworthiness through judge model selection and calibration.
• Assess retrieval and agents for grounding, tool-use validation, multi-step reasoning and failure recovery.
• Create Arabic golden sets and judges calibrated for Arabic; execute online evaluation and drift detection.
• Integrate evaluation and test automation as CI/CD quality gates.

Skills

Python (Advanced)

Leadership

LLM Evaluation

CI/CD Design

TypeScript

Statistics

English (B2+)

Arabic Evaluation

Job description

EPAM Systems in Turkey seeks an AI Solution/Principal Engineer to lead evaluation of a sovereign, multi-tenant AI platform and its applied products operating in Arabic and English. You will own how we measure and improve non-deterministic AI across assistants, retrieval, tools, and infrastructure, shaping products with trustworthy, evidence-based results.

Responsibilities include building evaluation harnesses, dataset versioning, online drift detection, and CI/CD integrations to gate quality