Senior Artificial Intelligence Evaluation Engineer

Description Ciklum; United States; On-site; USD 120,000 - 190,000; Full time

Job summary

Ciklum is seeking a Senior AI Evaluation Engineer to join our US-based team as a full-time role. You will work within a cross-functional product engineering group to advance AI evaluation across healthcare and safety-critical domains.

Responsibilities include building the central eval harness, creating golden datasets, setting pass thresholds, auditing pod evals, and monitoring drift in production. You will collaborate with experts to ensure robust, scalable AI evaluations and clear reporting.

Qualifications

• 4+ years in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering.
• Experience with eval frameworks and LLM-as-judge patterns.
• Statistical rigor on small samples and pass/fail thresholds.
• Healthcare or safety-critical evaluation experience; red-teaming background.

Responsibilities

• Build the central eval harness and templates; hub-and-spoke standards.
• Create golden datasets with business and clinical reviewers.
• Set and defend pass thresholds; audit evals and report results monthly.
• Stand up production monitoring for drift and regression in the pipeline.

Skills

ML/LLM evaluation

QA engineering for AI systems

Applied research engineering

Eval frameworks

Pass/fail thresholds

Drift monitoring

Tools

Eval frameworks

Test harnesses

Monitoring dashboards

Job description

Senior Artificial Intelligence Evaluation Engineer

Job Description

Ciklum is looking for a Senior AI Evaluation Engineer to join our team full-time in the US .

We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live.

About the role:

As a Senior AI Evaluation Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.

Responsibilities:

• Build the central eval harness and templates every pod uses; hub-and-spoke: central standards, pod-written tests
• Create golden datasets with business and clinical reviewers, including safety and medical-accuracy suites
• Set and defend pass thresholds; audit pod evals; report eval pass rates and incidents monthly
• Stand up production monitoring for drift and regression with the pipeline engineer

Requirements:

• 4+ years in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering
• Hands-on with eval frameworks, LLM-as-judge patterns and their failure modes, statistical rigor on small samples
• Independent spine: the team shipping a thing does not set its own pass bar
• Healthcare or safety-critical evaluation experience; red-teaming background

What's in it for you?

• Strong community: Work alongside top professionals in a friendly, open-door environment
• Growth focus: Take on large-scale projects with a global impact and expand your expertise
• Tailored learning: Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications
• Endless opportunities: Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies
• Care: Healthcare, Basic Life Insurance, Short and Long-term disability insurance according to the Company's Benefit Plans

About us:

At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, you'll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress. In the US, Ciklum is growing fast-inviting experienced professionals to lead digital transformation alongside Fortune 500 clients. Be part of a company where innovation and impact go hand in hand.

Explore, empower, engineer with Ciklum!

Required Skills

• AI Platform