Remote AI Observability Engineer (LLM)

Honeycomb.io; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Equity stake

Unlimited PTO

Remote-first culture

Home office stipend

Full benefits coverage

Job summary

Honeycomb is hiring a software engineer to deliver AI observability capabilities end-to-end, from ingestion to in-product visualizations, in a fully distributed team. You will work on data models, tooling, and UI with Go/Python and TS/React, collaborating with product managers and designers, and joining a 90-day on-call rotation.

The role offers a competitive base salary, equity, unlimited PTO, and comprehensive benefits within Honeycomb's remote-first culture.

Qualifications

• 3+ years of software engineering experience.
• Experience shipping customer-facing features at a SaaS company.
• Frontend and backend development with Go/Python and TS/React.
• Experience collaborating with product managers and designers.
• Regular, hands-on use of AI tools in day-to-day engineering work.
• Experience being on call (rotation after 90 days).
• Bonus: Experience building LLM/agent-powered features in production.
• Bonus: Observability tooling such as OpenTelemetry.

Responsibilities

• Deliver AI observability capabilities end to end, from ingestion to in-product visualizations.
• Contribute to the data model and infrastructure to observe agents and ecosystems across Honeycomb product.
• Build systems that analyze and surface AI-specific signals (cost, token usage, errors, anomalies).
• Collaborate with product and design to ensure usable, intuitive features and workflows.
• Share learnings about AI system behavior across conversations and workloads.
• Iterate with fast feedback loops and experimentation, sharing wins and failures.

Skills

Go

Python

TypeScript/React

AI tools usage

On-call experience

Tools

OpenTelemetry

Job description

Honeycomb is hiring a software engineer to deliver AI observability capabilities end-to-end, from ingestion to in-product visualizations, in a fully distributed team. You will work on data models, tooling, and UI with Go/Python and TS/React, collaborating with product managers and designers, and joining a 90-day on-call rotation.

The role offers a competitive base salary, equity, unlimited PTO, and comprehensive benefits within Honeycomb's remote-first culture.