Remote Senior SWE - LLM Observability & AI Signals

Candidate; United States; Remote; USD 183,000 - 206,000; Full time

Benefits offered by this job

Equity

Unlimited PTO

Home office stipend

Full benefits coverage

Parental leave

Development allowance

Job summary

Honeycomb is hiring for an engineer to own end-to-end AI observability features, from data ingestion to in-product visualizations. You will architect the data model and infrastructure to observe agents across the Honeycomb product, and build systems that surface AI signals like cost, tokens, errors, and anomalies.

You will collaborate with product managers and designers to ensure usable, intuitive workflows, and you’ll share best practices for exploring AI system behavior across conversations

Qualifications

• End-to-end development experience shipping features, especially in SaaS.
• Experience shipping customer-facing features beyond internal tooling.
• Strong full-stack capabilities: backend in Go/Python and frontend in TypeScript/React.

Responsibilities

• Own end-to-end development of AI observability capabilities, from ingestion to in-product visualizations.
• Architect and evolve the data model and infrastructure to observe agents and their ecosystems across the product.
• Build systems that analyze and surface AI-specific signals, such as cost, token usage, errors, and anomalies.
• Collaborate with product managers and designers to ensure usable, intuitive workflows.
• Teach best practices for exploring and understanding AI system behavior across conversations and workloads.
• Iterate through fast feedback loops and experiments, sharing wins and failures.

Skills

Go

Python

TypeScript

React

SaaS experience

PM collaboration

AI tooling

On-call

Ambiguity handling

LLM features

OpenTelemetry

Tools

OpenTelemetry

Job description

Honeycomb is hiring for an engineer to own end-to-end AI observability features, from data ingestion to in-product visualizations. You will architect the data model and infrastructure to observe agents across the Honeycomb product, and build systems that surface AI signals like cost, tokens, errors, and anomalies.

You will collaborate with product managers and designers to ensure usable, intuitive workflows, and you’ll share best practices for exploring AI system behavior across conversations