Senior Software Engineer, Agent Oversight & Observability

Scale AI, Inc.

San Francisco (CA)

On-site

USD 216,000 - 270,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity
Health, dental & vision coverage
Generous PTO
Learning & development stipend
Commuter stipend

Job summary

Scale AI, Inc. is seeking a Software Engineer on Agent Oversight to build platform infrastructure for observing, evaluating, and improving production agentic systems at scale.

You will develop observability tooling, evaluation harnesses, and data pipelines while maintaining high reliability and performance. You will collaborate with ML engineers and other teams to translate real-world deployment requirements into robust platform features, delivering end-to-end solutions from design to test and

Qualifications

  • 4+ years of professional software engineering experience with backend/distributed systems, APIs, and data pipelines.
  • Hands-on experience with production software for ML/LLM-powered products or platforms (evaluation systems, observability, experimentation infra).
  • Familiarity with production ML/LLM behavior, evaluation signals, and model-improvement workflows.
  • Experience partnering with ML engineers to build reliable platform capabilities and tools.

Responsibilities

  • Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production.
  • Build APIs and data pipelines to capture telemetry, evaluation signals, and performance metrics at scale.
  • Collaborate with ML engineers to ensure platform features support evaluation and improvement loops.
  • Own reliability, scalability, and observability of platform components for multiple enterprise customers.
  • Translate deployment requirements into platform features working with product and customers.
  • Develop end-to-end features: design, implementation, debugging, and testing.
  • Contribute to high-velocity experiments to validate platform capabilities against real usage.

Skills

Backend systems
APIs
Data pipelines
Observability
ML/LLM production knowledge
Platform ownership
Cross-functional collaboration
Problem solving

Job description

Scale AI, Inc. is seeking a Software Engineer on Agent Oversight to build platform infrastructure for observing, evaluating, and improving production agentic systems at scale.

You will develop observability tooling, evaluation harnesses, and data pipelines while maintaining high reliability and performance. You will collaborate with ML engineers and other teams to translate real-world deployment requirements into robust platform features, delivering end-to-end solutions from design to test and

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Platform Engineer, Agent Oversight & Observability
Senior Platform Engineer, Agent Oversight & Observability

Scale • San Francisco (CA)

On-site
USD 216,000 - 270,000
Health, dental, vision
Retirement plan
Learning stipend
+2
Senior AI Platform Engineer: Observability & Evaluation
Senior AI Platform Engineer: Observability & Evaluation

FDE Times • New York (NY)

On-site
USD 216,000 - 270,000
Equity compensation
Health insurance
Retirement benefits
+3
Engineering Manager, Agent Oversight & Production Platform
Engineering Manager, Agent Oversight & Production Platform

Scale AI, Inc. • San Francisco (CA), New York (NY)

On-site
USD 200,000 - 280,000
Agent Platform Engineer — AI Orchestration & Observability
Agent Platform Engineer — AI Orchestration & Observability

News Break • Mountain View (CA)

On-site
USD 120,000 - 220,000
Health, dental, and vision care forYou
401(K) matching program
Paid time off and holidays
+1
AI Agent Engineer for Observability (Remote)
AI Agent Engineer for Observability (Remote)

clickhouse • San Francisco (CA)

Remote
USD 150,000 - 190,000
Flexible work environment
Healthcare
Equity in the company
+3
Backend Engineer, Agent Platform (Production-Scale)
Backend Engineer, Agent Platform (Production-Scale)

ResumeXrays LLC • San Francisco (CA)

Hybrid
USD 180,000 - 240,000
Relocation assistance
Hybrid work model (3 days in office)
Senior AI Platform Engineer – Agent Infrastructure
Senior AI Platform Engineer – Agent Infrastructure

MaintainX • Austin (TX)

On-site
USD 130,000 - 236,000
AI Platform Engineer: Scalable Backends & AI Observability
AI Platform Engineer: Scalable Backends & AI Observability

Traversal • New York (NY)

On-site
USD 150,000 - 300,000
Health insurance
Startup equity
Senior Software Engineer, AI Agents & Platform Reliability
Senior Software Engineer, AI Agents & Platform Reliability

Fieldguide • San Francisco (CA)

On-site
USD 225,000 - 250,000
Competitive equity
Health benefits
401(k) plan
+1
Staff Software Engineer, AI Agent Observability
Staff Software Engineer, AI Agent Observability

Cisco Systems, Inc • New York (NY)

On-site
USD 180,000 - 240,000
Medical Insurance
Dental Insurance
Vision Insurance
+16