Senior Software Engineer, AI Evals

Sentry

San Francisco (CA)

On-site

USD 155,000 - 400,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Sentry is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure that measures accuracy, reliability, and real-world performance of our AI systems. You’ll help ensure debugging agents and AI features behave safely as we scale.

You’ll design datasets, golden test cases, and benchmarks from production data, create automated test harnesses and metrics pipelines, and translate product goals into concrete tests that drive higher confidence.

Qualifications

  • 5+ years of professional software experience in CS/ML.
  • Experience building testing, evaluation, or data infrastructure for AI/ML systems.
  • Proficiency in production-quality Python and TypeScript.
  • Experience with structured and unstructured datasets and data pipelines.
  • Familiarity with ML evaluation techniques and offline/online metrics.
  • Bonus: experience evaluating LLMs or AI-assisted developer tools.

Responsibilities

  • Design evaluation frameworks for AI systems.
  • Create datasets, golden test cases, and benchmarks from production data.
  • Build automated test harnesses and metrics pipelines.
  • Define good criteria with AI engineers and product leaders.
  • Own the evaluation lifecycle from experimentation to production monitoring.

Skills

AI/ML testing infra
Production-grade Python/TS
Structured data handling
Unstructured data workflows
ML evaluation techniques

Education

Bachelor's degree in CS/ML or related field

Tools

Python
TypeScript

Job description

About Sentry

Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building.

Trusted by 200,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future.

About the role

As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.

In this role you will
  • Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
  • Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
  • Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
  • Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
  • Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
You’ll love this job if you
  • Care deeply about correctness, rigor, and measurement in AI systems
  • Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
  • Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
  • Thrive in cross-functional environments and enjoy influencing model design through better evaluation
Qualifications
  • Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
  • Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
  • Comfort writing production-quality code (we use Python and TypeScript)
  • Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
  • Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
  • Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

The base salary range (or hourly wage range, if applicable) that Sentry reasonably expects to pay for this position is $155,000 to $400,000 USD. A successful candidate’s actual base salary (or hourly wage) amount will be determined by a variety of relevant factors including, without limitation, the candidate’s work location, education, work and other relevant experience, skills, and job-related knowledge. A successful candidate will be eligible to participate in Sentry’s employee benefit plans/programs applicable to the candidate’s position (including incentive compensation, equity grants, paid time off, and group health insurance coverage). See Sentry Benefits for more details about the Company’s benefit plans/programs.

Equal Opportunity at Sentry

Sentry is committed to providing equal employment opportunities to its employees and candidates for employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally-protected characteristic. This commitment includes the provision of reasonable accommodations to employees and candidates for employment with physical or mental disabilities who require such accommodations in order to (a) perform the essential functions of their jobs, or (b) seek employment with Sentry. We strive to build a diverse team, with an inclusive culture where every teammate can thrive. Sentry is an open-source company because we believe that everyone, everywhere, should have the ability and tools to make great software. Software should be accessible. That starts with making our industry accessible.

If you need assistance or an accommodation due to a disability, you may contact us at accommodations@sentry.io.

Want to learn more about how Sentry handles applicant data? Get the details in our Applicant Privacy Policy.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, AI
Senior Software Engineer, AI

Sentry • San Francisco (CA)

Hybrid
USD 190,000 - 280,000
Senior Software Engineer, AI Platform
Senior Software Engineer, AI Platform

Sentry.io • San Francisco (CA)

On-site
USD 155,000 - 400,000
Staff Software Engineer, AI Platform
Staff Software Engineer, AI Platform

Sentry • San Francisco (CA)

On-site
USD 240,000 - 400,000
Commuter Stipend
Professional Development
Flexible PTO
+3
Engineering Manager, Data Platform
Engineering Manager, Data Platform

Sentry • San Francisco (CA)

On-site
USD 220,000 - 400,000
Software Engineer, New Grad (2027)
Software Engineer, New Grad (2027)

Sentry • San Francisco (CA)

On-site
USD 145,000 - 150,000
Staff Software Engineer, Platform & Infrastructure
Staff Software Engineer, Platform & Infrastructure

Sentry • San Francisco (CA)

On-site
USD 240,000 - 400,000
Engineering Manager, Data Platform
Engineering Manager, Data Platform

Warmada • San Francisco (CA)

On-site
USD 220,000 - 400,000
Senior Staff Software Engineer, Platform
Senior Staff Software Engineer, Platform

Sentry • San Francisco (CA)

On-site
USD 260,000 - 400,000
Senior Manager, Support Engineering
Senior Manager, Support Engineering

Sentry • San Francisco (CA)

On-site
USD 160,000 - 180,000
Commuter Stipend
Professional Development Stipend
Flexible PTO
+2
Senior Technical Program Manager, AI Platform
Senior Technical Program Manager, AI Platform

Sentry • San Francisco (CA)

On-site
USD 150,000 - 240,000
Commuter Stipend
Professional Development Stipend
Flexible PTO
+2