LLM Observability Engineer - AI SaaS, Remote

Candidate; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Equity program

Unlimited PTO

Home office stipend

Full benefits coverage

Parental leave

Job summary

Honeycomb is a fully distributed company building AI observability tools and shipping features that let customers see how agents behave, how much they cost, and where errors occur. The team is small, cross-functional, and focused on AI system understanding across conversations and workloads.

We offer remote-friendly roles with transparent leveling and competitive compensation. Base salary ranges from $160,000 to $180,000 USD, plus equity and comprehensive benefits, with unlimited PTO and a home

Qualifications

• 3+ years of software engineering experience.
• Experience shipping customer-facing SaaS features.
• Full-stack proficiency: backend (Go/Python) and frontend (TypeScript/React).
• Collaborate with product managers and designers.
• Experience being on call after ~90 days.
• Navigate ambiguity in a fast-moving field with evolving requirements.
• Bonus: experience building LLM/agent-powered features in production.
• Bonus: OpenTelemetry observability tooling experience.

Responsibilities

• Deliver AI observability capabilities end to end, from ingestion to in-product visualizations.
• Contribute to the data model and infrastructure that observe agents and ecosystems across the Honeycomb product.
• Build systems that analyze and surface AI-specific signals such as cost, token usage, errors, and anomalies.
• Collaborate with product and design partners to ensure usability and meaningful user workflows.
• Share what you learn about exploring and understanding AI system behavior across conversations and workloads.
• Iterate through fast feedback loops and experimentation, and share both wins and failures.

Skills

Software engineering

Backend development

Frontend development

AI tooling usage

On-call experience

Ambiguity navigation

Tools

Go

Python

TypeScript

React

OpenTelemetry

LLM features

Job description

Honeycomb is a fully distributed company building AI observability tools and shipping features that let customers see how agents behave, how much they cost, and where errors occur. The team is small, cross-functional, and focused on AI system understanding across conversations and workloads.

We offer remote-friendly roles with transparent leveling and competitive compensation. Base salary ranges from $160,000 to $180,000 USD, plus equity and comprehensive benefits, with unlimited PTO and a home