Lead AI Observability Engineer (LLM Systems)

Datadog, Inc.; Boston, New York (MA, NY); Hybrid; USD 234,000 - 300,000; Full time

Benefits offered by this job

Competitive global benefits

Influence product direction

Collaborative team

Hybrid work model

Job summary

Datadog is seeking a Staff Engineer for the Agent Observability team to lead the development of LLM observability features and products. You’ll shape direction, drive experiments, and apply deep AI and software engineering knowledge to real-world, production-grade systems.

You’ll mentor engineers, collaborate across engineering, product, UX, and applied science, and stay close to customer pain points to prioritise impactful work in a hybrid Datadog environment.

Qualifications

• BS/MS/PhD in Computer Science, Engineering or related field or equivalent experience.
• Deep understanding of distributed systems and scalable backend architectures.
• Hands-on experience building and shipping LLM-powered or GenAI applications.
• Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering.
• Ability to thrive in ambiguous, fast-changing spaces with a product-oriented mindset.

Responsibilities

• Drive design and implementation of LLM observability features.
• Ideate, prototype, and scale new product features to provide insights for generative AI systems.
• Work cross-functionally with eng, product, UX, and applied science to iterate fast and find product-market fit.
• Develop and extend tools for tracing, evaluating, and debugging LLMs.
• Influence architecture decisions and mentor engineers to build resilient, high-performance systems.
• Stay close to customer pain points to guide product and engineering priorities.
• Stay current with AI observability trends and drive team innovation.

Skills

LLM observability

Distributed systems

GenAI applications

Prompt engineering

Mentoring engineers

Product mindset

Observability tools

Education

BS/MS/PhD in CS or related field

Job description

Datadog is seeking a Staff Engineer for the Agent Observability team to lead the development of LLM observability features and products. You’ll shape direction, drive experiments, and apply deep AI and software engineering knowledge to real-world, production-grade systems.

You’ll mentor engineers, collaborate across engineering, product, UX, and applied science, and stay close to customer pain points to prioritise impactful work in a hybrid Datadog environment.