Staff AI Observability & Telemetry Engineer

Bitdeer Technologies Group; San Jose (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Bitdeer is seeking a Staff AI Observability & Telemetry Engineer to architect the telemetry backbone of its NeoCloud platform. You will collect, store, and analyze millions of signals per second to support SREs, automated remediation agents, and external customers in monitoring GPU workloads and network performance.

You will define telemetry standards, build high-fidelity dashboards, and lead design reviews, collaborating with GPU Systems, Scheduling, and ML teams to ensure scalable,

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related field.
• 6+ years of software or site reliability engineering, with deep, hands-on expertise in Prometheus/OpenTelemetry.
• Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators.
• Hands-on experience with kernel-level tracing tools (eBPF, BCC) and deep performance tuning of Linux systems.
• Strong familiarity with AI hardware metrics (GPU power states, SM utilization, memory bandwidth) and high-performance network telemetry.
• Proven track record of operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments.
• Strong technical leadership skills; ability to influence architectural decisions and align cross-functional teams around observability standards.
• Excellent communication skills, with the ability to translate complex system requirements into manageable engineering milestones.
• Experience working in high-velocity, high-growth engineering environments is strongly preferred.

Responsibilities

• Architect and scale a high-cardinality telemetry infrastructure using highly available time-series databases.
• Integrate hardware-level exporters directly into the Kubernetes observability stack for a unified cluster view.
• Build eBPF-based diagnostic tools to trace network congestion and kernel I/O latency.
• Develop automated dashboards and alerting pipelines for proactive cordoning of degraded hardware.
• Design metric pipelines for real-time GPU and network utilization-based billing.
• Collaborate with GPU Systems and Scheduling to set observability standards for AI-native workloads.
• Lead design reviews for observability architecture and mentor on high-performance telemetry collection.

Skills

Go

Prometheus

OpenTelemetry

Kubernetes

eBPF

Linux performance

GPU telemetry

Telemetry pipelines

Leadership

Education

BSc or MSc in CS/EE

Tools

VictoriaMetrics

Thanos

Mimir

NVIDIA DCGM

IPMI

Redfish

Job description

Bitdeer is a world-leading technology company for AI and Bitcoin mining infrastructure.

Bitdeer is committed to providing comprehensive Bitcoin mining solutions for its customers and building AI computational infrastructure to support the AI revolution. Bitdeer handles complex processes involved in computing such as equipment procurement, transport logistics, data center design and construction, equipment management, and daily operations. Bitdeer also offers advanced cloud capabilities to customers with high demand for artificial intelligence.

Headquartered in Singapore, Bitdeer has deployed data centers across multiple countries, including the United States, Norway, Bhutan, and Ethiopia.Bitdeer also offers advanced cloud capabilities to customers with high demand for artificial intelligence. https://ir.bitdeer.com/

Position Overview

We are seeking a Staff AI Observability & Telemetry Engineer to architect the "nervous system" of our AI-native NeoCloud platform. This role goes beyond standard monitoring; you are responsible for building the high-fidelity perception layer required to orchestrate massive-scale AI infrastructure. You will capture, store, and make sense of millions of hardware and software signals per second, enabling our SREs, automated remediation agents, and external customers to peer deep into the performance of their GPU workloads and the underlying network fabric. You will define the telemetry standards that drive our autonomous operations, ensuring we can detect, diagnose, and resolve hardware and software bottlenecks in real-time.

Key Responsibilities

• Architect and scale a high-cardinality telemetry infrastructure using highly available time-series databases (e.g., VictoriaMetrics, Thanos, or Mimir) capable of handling massive ingestion rates.
• Integrate complex hardware-level exporters (NVIDIA DCGM, network switch telemetry, IPMI/Redfish) directly into the Kubernetes observability stack to provide a unified view of the cluster.
• Build eBPF-based diagnostic tools to trace network congestion, kernel-level I/O latency, and distributed training bottlenecks across the cluster.
• Develop automated dashboards and alerting pipelines that trigger proactive cordoning of degraded hardware before it impacts customer training jobs.
• Design the metric pipelines required for accurate, multi-tenant consumption billing based on real-time GPU and network utilization metrics.
• Collaborate with the GPU Systems and Scheduling teams to create observability standards for "AI-native" workloads, ensuring deep insight into job efficiency and resource utilization.
• Lead technical design reviews for observability architecture, mentoring team members on best practices for high-performance telemetry collection and analysis.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related field.
• 6+ years of software or site reliability engineering, with deep, hands-on expertise in the Prometheus/OpenTelemetry ecosystem.
• Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators.
• Hands-on experience with kernel-level tracing tools (eBPF, BCC) and deep performance tuning of Linux systems.
• Strong familiarity with AI hardware metrics (GPU power states, SM utilization, memory bandwidth) and high-performance network telemetry.
• Proven track record of operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments.
• Strong technical leadership skills; ability to influence architectural decisions and align cross-functional teams around observability standards.
• Excellent communication skills, with the ability to translate complex system requirements into manageable engineering milestones.
• Experience working in high-velocity, high-growth engineering environments is strongly preferred.

--------------------------------------------------------------------

Bitdeer is committed to providing equal employment opportunities in accordance with country, state, and local laws. Bitdeer does not discriminate against employees or applicants based on conditions such as race, color, gender identity and/or expression, sexual orientation, marital and/or parental status, religion, political opinion, nationality, ethnic background or social origin, social status, disability, age, indigenous status, and union.