AI Solution/Principal Engineer

EPAM Systems; Turkey; On-site; TRY 5,938,000 - 9,401,000; Full time

Benefits offered by this job

Private health insurance

Career development

Mentoring programs

Learning platform access

English courses

Job summary

EPAM Systems in Turkey seeks an AI Solution/Principal Engineer to lead evaluation of a sovereign, multi-tenant AI platform and its applied products operating in Arabic and English. You will own how we measure and improve non-deterministic AI across assistants, retrieval, tools, and infrastructure, shaping products with trustworthy, evidence-based results.

Responsibilities include building evaluation harnesses, dataset versioning, online drift detection, and CI/CD integrations to gate quality

Qualifications

• 5+ years building evaluation or test infrastructure and shared libraries.
• Leadership experience of at least 1 year.
• Expertise in AI evaluation spanning output quality, retrieval and grounding.
• Proficiency in Python; intermediate in TypeScript.
• Statistics knowledge for non-deterministic systems.

Responsibilities

• Define evaluation strategy: what gets measured, layers, methodology, and feed product decisions.
• Build evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection and reporting.
• Ensure grading trustworthiness through judge model selection and calibration.
• Assess retrieval and agents for grounding, tool-use validation, multi-step reasoning and failure recovery.
• Create Arabic golden sets and judges calibrated for Arabic; execute online evaluation and drift detection.
• Integrate evaluation and test automation as CI/CD quality gates.

Skills

Python (Advanced)

Leadership

LLM Evaluation

CI/CD Design

TypeScript

Statistics

English (B2+)

Arabic Evaluation

Job description

We are seeking an AI Solution/Principal Engineer to join a team building a sovereign, multi-tenant agentic AI platform together with the applied products running on top of it. Every component ships under strict data residency constraints and operates in both Arabic and English. Our squads are small and senior, and we need someone who owns how the organization knows its AI systems perform — evaluation across assistants, retrieval pipelines, agent workflows, voice and document systems, and the infrastructure that transforms \"it seems better\" into evidence someone can act on.This is not a QA role with AI added. Test automation and performance testing are in scope, but the center of gravity is evaluating non-deterministic systems, where the same input produces different outputs, \"correct\" is often fuzzy, and failure modes such as hallucination, grounding failure, drift, and prompt injection are ones no assertion library catches.ResponsibilitiesDefine evaluation strategy: what gets measured, at which layer, with what methodology, and how results feed product decisionsBuild the shared platform, covering evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection, and reporting leadership can readEnsure grading is trustworthy through judge model selection, rubric design, calibration against human labels, and knowing when automated grading cannot be trustedAssess retrieval and agents for grounding and citation correctness, tool-use validation, multi-step reasoning and failure recoveryCreate Arabic golden sets and judges calibrated for Arabic rather than assumed to transfer from EnglishExecute online evaluation and drift detectionPerform prompt injection, jailbreak and data-leakage red-teamingIntegrate evaluation and test automation together as quality gates in CI/CDRequirements5+ years of experience building evaluation or test infrastructure others depend on, including harnesses, shared libraries and frameworksAt least 1 year of relevant leadership experienceExpertise in AI evaluation spanning output quality, retrieval and grounding, and regression detection on non-deterministic behaviorProficiency in LLM-as-judge methodology: rubric design, calibration against human labels, and understanding of its failure modesProficiency in Python at an advanced level, deep enough to build shared libraries, with intermediate competency in TypeScript or JavaKnowledge of statistics for non-deterministic systems, covering sampling, confidence intervals, inter-rater agreement and significanceSkills in CI/CD framework design across API, web and data surfaces, with test selection, parallelization and flake managementBackground in golden sets and dataset versioningProficiency in English at an Upper-Intermediate level (B2) or higherNice to haveFamiliarity with ragas, DeepEval, Promptfoo, Braintrust, LangSmith or Langfuse toolsExpertise in Arabic evaluation, spanning golden sets, dialect coverage, right-to-left validation and judge calibrationBackground in adversarial and security testing, including red-teaming practicesCapability to carry out voice and conversational evaluation, plus performance testing against AI servicesShowcase of delivery in a regulated or government environmentWe offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.