Senior AI Evaluation Engineer: ML Infra & Benchmarks

Sentry

San Francisco (CA)

On-site

USD 155,000 - 400,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Sentry is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure that measures accuracy, reliability, and real-world performance of our AI systems. You’ll help ensure debugging agents and AI features behave safely as we scale.

You’ll design datasets, golden test cases, and benchmarks from production data, create automated test harnesses and metrics pipelines, and translate product goals into concrete tests that drive higher confidence.

Qualifications

  • 5+ years of professional software experience in CS/ML.
  • Experience building testing, evaluation, or data infrastructure for AI/ML systems.
  • Proficiency in production-quality Python and TypeScript.
  • Experience with structured and unstructured datasets and data pipelines.
  • Familiarity with ML evaluation techniques and offline/online metrics.
  • Bonus: experience evaluating LLMs or AI-assisted developer tools.

Responsibilities

  • Design evaluation frameworks for AI systems.
  • Create datasets, golden test cases, and benchmarks from production data.
  • Build automated test harnesses and metrics pipelines.
  • Define good criteria with AI engineers and product leaders.
  • Own the evaluation lifecycle from experimentation to production monitoring.

Skills

AI/ML testing infra
Production-grade Python/TS
Structured data handling
Unstructured data workflows
ML evaluation techniques

Education

Bachelor's degree in CS/ML or related field

Tools

Python
TypeScript

Job description

Sentry is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure that measures accuracy, reliability, and real-world performance of our AI systems. You’ll help ensure debugging agents and AI features behave safely as we scale.

You’ll design datasets, golden test cases, and benchmarks from production data, create automated test harnesses and metrics pipelines, and translate product goals into concrete tests that drive higher confidence.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, AI Evals
Senior Software Engineer, AI Evals

Sentry • San Francisco (CA)

On-site
USD 155,000 - 400,000
Senior AI Platform Engineer – Build Scalable Agent Systems
Senior AI Platform Engineer – Build Scalable Agent Systems

Sentry.io • San Francisco (CA)

On-site
USD 155,000 - 400,000
Staff AI Platform Engineer: Build Next-Gen Agentic AI
Staff AI Platform Engineer: Build Next-Gen Agentic AI

Sentry • San Francisco (CA)

On-site
USD 240,000 - 400,000
Commuter Stipend
Professional Development
Flexible PTO
+3
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
ML Engineer: Build Real-World AI Evaluation Pipelines
ML Engineer: Build Real-World AI Evaluation Pipelines

Speedrun Talent Network • San Francisco (CA)

Hybrid
USD 150,000 - 210,000
Competitive compensation
Health and wellness benefits
Cutting-edge AI projects
+1
Senior AI Evaluation & Benchmark Engineer (Hybrid)
Senior AI Evaluation & Benchmark Engineer (Hybrid)

Glean • San Francisco (CA)

Hybrid
USD 200,000 - 280,000
Medical, Vision, Dental
Generous time-off
401k plan
+4
ML Engineer - AI Evaluation & Benchmarking Platform
ML Engineer - AI Evaluation & Benchmarking Platform

Arena Intelligence, Inc. • California

On-site
USD 170,000 - 230,000
Comprehensive health benefits
Cutting-edge AI projects
Small, mission-driven team
Senior Manager, AI-Driven Support Engineering
Senior Manager, AI-Driven Support Engineering

Sentry • San Francisco (CA)

On-site
USD 160,000 - 180,000
Commuter Stipend
Professional Development Stipend
Flexible PTO
+2
ML Engineer — Core AI Evaluation Platform
ML Engineer — Core AI Evaluation Platform

Arena Intelligence, Inc. • San Francisco (CA)

On-site
USD 140,000 - 210,000
Competitive compensation
Health and wellness benefits
Cutting-edge AI projects
+1
Senior AI Systems Engineer - Evaluation & Reliability
Senior AI Systems Engineer - Evaluation & Reliability

ExpertsHub.ai • San Francisco (CA)

On-site
USD 250,000 - 360,000
Equity
Medical Insurance
Dental Insurance
+4