Staff ML Observability Engineer (LLM Production)

Datadog; New York (NY); Hybrid; USD 234,000 - 300,000; Full time

Benefits offered by this job

Hybrid work model

Competitive salary & equity

Healthcare benefits

Paid time off

Job summary

Datadog, the leading observability platform, seeks a Staff Engineer to lead LLM observability feature work and foundational capabilities. You will shape product direction, drive experimentation, and mentor engineers to build scalable, high-performance systems in production.

You will collaborate across engineering, product, UX, and applied science to deliver tools that help customers monitor and optimize LLM-based applications, with a strong emphasis on reliability and observability.

Qualifications

• BS/MS/PhD in Computer Science or related field or equivalent experience.
• Deep understanding of distributed systems and scalable backend architectures.
• Hands-on experience building LLM-powered or GenAI applications.
• Experience with observability tools and platforms.
• Product-minded with ability to thrive in ambiguous spaces.

Responsibilities

• Drive design and implementation of LLM observability features.
• Ideate, prototype, and scale new product features for generative AI systems.
• Collaborate cross-functionally with engineering, product, UX, and applied science to iterate fast and find product-market fit.
• Develop and extend tools for tracing, evaluating, and debugging LLMs.
• Influence architecture decisions and mentor engineers to build resilient, high-performance systems.
• Stay close to customer pain points to guide product and engineering priorities.
• Stay current with industry trends and advancements in ML and observability.

Skills

Distributed systems

LLM/GenAI

Prompt engineering

Observability

Education

BS/MS/PhD in Computer Science or related field

Tools

Observability tools

Job description

Datadog, the leading observability platform, seeks a Staff Engineer to lead LLM observability feature work and foundational capabilities. You will shape product direction, drive experimentation, and mentor engineers to build scalable, high-performance systems in production.

You will collaborate across engineering, product, UX, and applied science to deliver tools that help customers monitor and optimize LLM-based applications, with a strong emphasis on reliability and observability.