Senior AI Observability Platform Engineer

Socket.dev; United States; On-site; USD 160,000 - 230,000; Full time

Job summary

Nscale is seeking a Senior Observability Platform Engineer to design, build, and scale our observability stack for GPU clusters and AI workloads. You’ll treat observability as a product, delivering reliable metrics, logs, and traces while reducing noise and improving debugging.

You’ll partner with SRE, infrastructure, and AI/ML teams to embed observability into services, develop reusable patterns, and influence platform direction through hands-on engineering and incident postmortems.

Qualifications

• 5+ years in SRE or observability roles.
• Experience operating production observability systems.
• Proficient in Python or Go.
• Experience with Kubernetes-based infrastructure.

Responsibilities

• Design, build, and operate scalable observability systems across metrics, logs, traces, and alerting.
• Contribute to architectural decisions around tooling, data pipelines, storage, and retention.
• Improve signal quality by reducing noise and refining alerting practices.
• Identify observability gaps and address reliability impacts.
• Collaborate with SRE, infrastructure, and AI/ML teams to embed observability into services.

Skills

SRE experience

Observability

Production systems

Python or Go

Tools

Kubernetes

Prometheus

Thanos

VictoriaMetrics

Grafana

Loki

OpenTelemetry

ClickHouse

Elastic

Terraform

Ansible

Job description

Nscale is seeking a Senior Observability Platform Engineer to design, build, and scale our observability stack for GPU clusters and AI workloads. You’ll treat observability as a product, delivering reliable metrics, logs, and traces while reducing noise and improving debugging.

You’ll partner with SRE, infrastructure, and AI/ML teams to embed observability into services, develop reusable patterns, and influence platform direction through hands-on engineering and incident postmortems.