Senior Observability Engineer: Telemetry for AI Systems

Anthropic Limited; Seattle (WA); Hybrid; USD 320,000 - 485,000; Full time

Benefits offered by this job

Equity donation matching

Generous vacation & parental leave

Flexible working hours

Office in SF

Job summary

Anthropic is seeking Software Engineers to join the Observability team within the Infrastructure organization. You will design and build scalable telemetry pipelines and instrumentation libraries to monitor large GPU, TPU, and Trainium clusters, helping engineers observe system behavior across the fleet.

You will own core observability platforms, reduce MTTR, and collaborate with Research, Inference, Product, and Infrastructure teams to deliver high-quality telemetry, tracing, profiling, and

Qualifications

• Experience building large-scale observability infrastructure.
• End-to-end telemetry signals from instrumentation through ingest to query and analysis.
• Comfort debugging below the application layer, including kernel and network stack.
• Excellent communication skills and ability to partner with internal teams.

Responsibilities

• Design and build scalable telemetry ingest and storage pipelines for metrics, logs, traces, and error data across multi-cluster infrastructure.
• Build observability solutions that give engineers deep visibility into system behavior across the fleet.
• Own and evolve core observability platforms, leading migrations and architectural improvements.
• Build instrumentation libraries and eBPF-based auto-instrumentation to emit high-quality telemetry.
• Reduce mean time to detection and resolution by cross-signal correlation and unified query interfaces.
• Turn continuous profiling and utilization telemetry into actionable optimization insights across CPU, memory, and accelerators.
• Collaborate with Research, Inference, Product, and Infrastructure teams to meet unique needs.

Skills

Observability architecture

Telemetry pipelines

eBPF instrumentation

Kernel-level debugging

Distributed tracing

OpenTelemetry

Education

Bachelor's degree in CS or related field

Tools

OpenTelemetry

ClickHouse

BigQuery

Telemetry pipelines

Job description

Anthropic is seeking Software Engineers to join the Observability team within the Infrastructure organization. You will design and build scalable telemetry pipelines and instrumentation libraries to monitor large GPU, TPU, and Trainium clusters, helping engineers observe system behavior across the fleet.

You will own core observability platforms, reduce MTTR, and collaborate with Research, Inference, Product, and Infrastructure teams to deliver high-quality telemetry, tracing, profiling, and