Senior AI Platform Engineer: Observability & Evaluation

FDE Times

New York (NY)

On-site

USD 216,000 - 270,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Health insurance
Retirement benefits
Learning stipend
Paid time off (PTO)
Commuter stipend

Job summary

Scale is seeking a Software Engineer for Agent Oversight to design platform infrastructure that lets production agents be observed, evaluated, and improved at scale. You will build observability tooling, evaluation harnesses, and data pipelines connecting to improvement loops, working closely with ML engineers to ensure robust workflows.

You will own the reliability, scalability, and observability of platform components used by enterprise and government customers, translating deployment needs

Qualifications

  • 4+ years of professional software engineering experience with backend/distributed systems, APIs, and data pipelines.
  • Hands-on experience building production software for ML/LLM-powered products or platforms.
  • Knowledge of how ML systems behave in production: evaluation signals, failure modes, prompt/tool-calling workflows.
  • Experience partnering with ML engineers to create reliable platform capabilities.
  • Experience building infrastructure that other teams build on top of.
  • Owner mindset for features from design to production.
  • Comfort in ambiguous, fast-changing domains.

Responsibilities

  • Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production.
  • Build reliable APIs and data pipelines that capture agent telemetry, evaluation signals, and performance metrics at scale.
  • Collaborate with ML engineers on evaluation or improvement systems, ensuring reliability of workflows.
  • Own the reliability, scalability, and observability of platform components serving enterprise and government customers.
  • Translate real-world deployment requirements into platform features in cross-functional teams.
  • Build features end-to-end: design, implementation, debugging, testing.
  • Participate in rapid experimentation to validate platform capabilities with real usage.

Skills

Backend systems
APIs
Data pipelines
Observability
ML/LLM familiarity
Platform engineering
Cross-functional collaboration
Problem solving
Mentoring

Tools

Observability tooling
Evaluation harnesses
Telemetry pipelines

Job description

Scale is seeking a Software Engineer for Agent Oversight to design platform infrastructure that lets production agents be observed, evaluated, and improved at scale. You will build observability tooling, evaluation harnesses, and data pipelines connecting to improvement loops, working closely with ML engineers to ensure robust workflows.

You will own the reliability, scalability, and observability of platform components used by enterprise and government customers, translating deployment needs

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, Agent Oversight & Observability
Senior Software Engineer, Agent Oversight & Observability

Scale AI, Inc. • San Francisco (CA)

On-site
USD 216,000 - 270,000
Equity
Health, dental & vision coverage
Generous PTO
+2
Senior Platform Engineer, Agent Oversight & Observability
Senior Platform Engineer, Agent Oversight & Observability

Scale • San Francisco (CA)

On-site
USD 216,000 - 270,000
Health, dental, vision
Retirement plan
Learning stipend
+2
AI Platform Engineer: Scalable Backends & AI Observability
AI Platform Engineer: Scalable Backends & AI Observability

Traversal • New York (NY)

On-site
USD 150,000 - 300,000
Health insurance
Startup equity
Senior GenAI Platform Engineer - Observability & Orchestration
Senior GenAI Platform Engineer - Observability & Orchestration

Ethereum Technologies • Chicago (IL)

On-site
USD 140,000 - 210,000
Engineering Manager, Agent Oversight & Production Platform
Engineering Manager, Agent Oversight & Production Platform

Scale AI, Inc. • San Francisco (CA), New York (NY)

On-site
USD 200,000 - 280,000
Senior AI Platform Engineer – Scale AI Observability
Senior AI Platform Engineer – Scale AI Observability

Coreweave • Sunnyvale (CA)

On-site
USD 165,000 - 220,000
Medical, dental, and vision insurance
Company-paid Life Insurance
401(k) with employer match
+3
Agent Platform Engineer — AI Orchestration & Observability
Agent Platform Engineer — AI Orchestration & Observability

News Break • Mountain View (CA)

On-site
USD 120,000 - 220,000
Health, dental, and vision care forYou
401(K) matching program
Paid time off and holidays
+1
Senior AI Platform Engineer - Observability & Agents
Senior AI Platform Engineer - Observability & Agents

Coreweave • Sunnyvale (CA)

On-site
USD 165,000 - 220,000
Medical, dental, and vision insurance
401(k) with employer match
Flexible PTO
+1
AI Platform Observability Engineer
AI Platform Observability Engineer

Moring AI • Alpharetta (GA)

On-site
USD 60,000 - 90,000
Stock options
Senior AI/ML Platform Engineer for Scalable Infra
Senior AI/ML Platform Engineer for Scalable Infra

Employee Relations Associates, Inc. • Denver (CO)

Hybrid
USD 170,000 - 210,000
Medical coverage
Dental coverage
Vision coverage
+4