Senior Software Engineer I - Agent Observability

honeycomb; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Equity

Unlimited PTO

Health coverage

Parental leave

Development allowance

Job summary

Honeycomb is seeking an engineer to build observability features that help users understand AI agents, their costs, and where they fail. Work spans from data models to customer-facing tools for exploring agent conversations and workloads.

You'll join a small cross-functional team and ship capabilities used in production, with on-call duties starting after ~90 days. Collaboration with product and design is essential to shape usable workflows for customers.

Qualifications

• 3+ years of software engineering experience, including shipping customer-facing SaaS features.
• Full-stack development with backend in Go and/or Python and frontend in TypeScript/React.
• Experience collaborating with product managers and designers.
• Regular hands-on use of AI tools in engineering work.
• Comfort with on-call support and changing requirements.

Responsibilities

• Develop AI observability features end to end, from data ingestion to product visualizations.
• Improve the data model and infrastructure for observing agents and their surrounding systems.
• Analyze and surface signals such as token usage, costs, errors, and unusual behavior.
• Partner with product and design to shape usable workflows around customer needs.
• Share findings about how users investigate AI behavior, and iterate through experiments and feedback.
• Participate in an on-call rotation, expected to begin after roughly 90 days.

Skills

Go

Python

TypeScript

React

On-call experience

Collaborate with product/design

Tools

OpenTelemetry

Job description

Role overview

Build observability features that help users understand how AI agents behave, what they cost, and where they fail. The work spans the underlying data model and ingestion systems through customer-facing tools for exploring agent conversations and workloads. You’ll work in a small cross-functional team and ship capabilities used in production.

Responsibilities

• Develop AI observability features end to end, from data ingestion to product visualizations.
• Improve the data model and infrastructure for observing agents and their surrounding systems.
• Analyze and surface signals such as token usage, costs, errors, and unusual behavior.
• Partner with product and design to shape usable workflows around customer needs.
• Share findings about how users investigate AI behavior, and iterate through experiments and feedback.
• Participate in an on-call rotation, expected to begin after roughly 90 days.

Requirements

• At least 3 years of software engineering experience, including shipping customer-facing SaaS features.
• Full-stack development skills, including backend work in Go and/or Python and frontend work in TypeScript and React.
• Experience collaborating with product managers and designers.
• Regular hands-on use of AI tools in engineering work.
• Experience with on-call support and comfort working through changing requirements and ambiguity.

Nice to have

• Experience building production features powered by large language models or AI agents.
• Familiarity with observability tools such as OpenTelemetry.

Benefits and work setup

• Fully distributed work, with home office, coworking, and internet stipends.
• Base salary of $160,000–$180,000 USD, depending on experience level, plus equity.
• Unlimited paid time off, health coverage, up to 16 weeks of paid parental leave, and an annual development allowance.