Senior AI Infra Observability Engineer

Togetherai; San Francisco (CA); On-site; USD 200,000 - 280,000; Full time

Benefits offered by this job

Health insurance

Startup equity

Remote work flexibility

Competitive salary

Job summary

Together AI is building the AI Acceleration Cloud. We seek an experienced observability and infrastructure engineer to design scalable monitoring, logging, and tracing for a distributed AI platform, enabling real-time insights and reliable operations.

Responsibilities include building automated alerts, SLIs/SLOs, and anomaly detection, plus deploying tools with Go, Python, Terraform, Ansible, and Helm. You will guide best practices and collaborate across teams on incident response and

Qualifications

• Expertise in observability platforms (Prometheus, Grafana, ClickStack, OpenTelemetry) and cloud-native monitoring services (AWS, GCP, Azure).
• Strong programming skills in Go, Python, or similar, with proficiency in IaC tools (Terraform, Ansible, Helm).
• Experience designing, operating, and scaling large distributed systems and pipelines for high-volume ingestion and real-time queries.
• Deep understanding of containerization (Docker) and orchestration (Kubernetes).
• Knowledge of microservices, service mesh, CI/CD, and GitOps workflows.
• Expertise with databases (PostgreSQL, MongoDB, Redis) and time-series DBs with high-cardinality data.

Responsibilities

• Design and implement a scalable observability platform (metrics, logs, traces) using Prometheus, Grafana, ClickHouse, ClickStack, and OpenTelemetry.
• Develop automated monitoring, alerting, and anomaly detection systems including SLIs/SLOs and runbooks.
• Build and deploy observability tooling and IaC using Go, Python, Terraform, Ansible, and Helm.
• Collaborate with engineering to enhance tracing and monitoring, and lead post-mortem incident analyses.
• Define observability best practices for the organization.

Skills

Go

Python

Cloud-native

Prometheus

Grafana

OpenTelemetry

Docker

Kubernetes

Tools

ClickStack

Terraform

Ansible

Helm

PostgreSQL

MongoDB

Redis

Job description

Together AI is building the AI Acceleration Cloud. We seek an experienced observability and infrastructure engineer to design scalable monitoring, logging, and tracing for a distributed AI platform, enabling real-time insights and reliable operations.

Responsibilities include building automated alerts, SLIs/SLOs, and anomaly detection, plus deploying tools with Go, Python, Terraform, Ansible, and Helm. You will guide best practices and collaborate across teams on incident response and