Senior Artificial Intelligence Evaluation Engineer

Ciklum; United States; On-site; USD 120,000 - 170,000; Full time

Job summary

Ciklum is seeking a Senior AI Evaluation Engineer in the United States to lead ML/LLM evaluation initiatives within a cross-functional team. You will design and maintain evaluation harnesses, develop gold datasets, and set thresholds to monitor model performance and safety at scale.

The role focuses on rigorous QA for AI systems, building robust evaluation pipelines, and collaborating with product and engineering teams to ensure reliability and safety across deployments.

Qualifications

• 4+ years in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering.
• Hands-on with eval frameworks and LLM-as-judge patterns.
• Experience with safety-critical evaluation and red-teaming is preferred.

Responsibilities

• Build the central eval harness and templates every pod uses; hub-and-spoke: central standards, pod-written tests.
• Create golden datasets with business and clinical reviewers, including safety and medical-accuracy suites.
• Set and defend pass thresholds; audit pod evals; report eval pass rates and incidents monthly.
• Stand up production monitoring for drift and regression with the pipeline engineer.

Skills

AI Platform

Tools

Eval frameworks

LLM-as-judge patterns

Job description

Senior Artificial Intelligence Evaluation Engineer

Job Description

Ciklum is looking for a Senior AI Evaluation Engineer to join our team full-time in the US .

We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live.

About the role:

As a Senior AI Evaluation Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.

Responsibilities:

• Build the central eval harness and templates every pod uses; hub-and-spoke: central standards, pod-written tests
• Create golden datasets with business and clinical reviewers, including safety and medical-accuracy suites
• Set and defend pass thresholds; audit pod evals; report eval pass rates and incidents monthly
• Stand up production monitoring for drift and regression with the pipeline engineer

Requirements:

• 4+ years in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering
• Hands-on with eval frameworks, LLM-as-judge patterns and their failure modes, statistical rigor on small samples
• Independent spine: the team shipping a thing does not set its own pass bar
• Healthcare or safety-critical evaluation experience; red-teaming background

What's in it for you?

• Strong community: Work alongside top professionals in a friendly, open-door environment
• Growth focus: Take on large-scale projects with a global impact and expand your expertise
• Tailored learning: Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications
• Endless opportunities: Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies
• Care: Healthcare, Basic Life Insurance, Short and Long-term disability insurance according to the Company's Benefit Plans

About us:

At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, you'll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress. In the US, Ciklum is growing fast-inviting experienced professionals to lead digital transformation alongside Fortune 500 clients. Be part of a company where innovation and impact go hand in hand.

Explore, empower, engineer with Ciklum!

Required Skills

• AI Platform