Senior Software Engineer II - LLM Observability

honeycomb; United States; Remote; USD 175,000 - 196,000; Full time

Benefits offered by this job

Fully distributed

Equity

Unlimited PTO

Home office stipend

Internet support

Parental leave

Development allowance

Job summary

Honeycomb is hiring for a full-stack role focused on observability features for AI agents in production. You will own features end-to-end, from data ingestion to in-product visuals, with a focus on token use, cost, errors, and behavior signals.

Join a distributed team and collaborate with product and design to deliver customer-centric workflows. A strong background in Go, Python, TypeScript, and React is expected, with on-call rotation experience beneficial.

Qualifications

• Significant software engineering experience with feature ownership from idea to delivery.
• Experience shipping customer-facing SaaS product features.
• Full-stack capability: Go and/or Python backend; TypeScript/React frontend.
• Strong collaboration with product managers and designers.
• Hands-on use of AI tools in engineering work.
• On-call rotation experience and comfort with changing requirements.

Responsibilities

• Develop AI observability features across the pipeline, from data ingestion to in-product visuals.
• Design and evolve data models and infrastructure for observing agents and systems.
• Build analyses and surfaces for token use, cost, errors, and unusual behavior.
• Collaborate with product and design to shape clear customer workflows.
• Help customers explore agent behavior across conversations and workloads.
• Share lessons from experiments and iterations to improve features.

Skills

Backend Go

Backend Python

Frontend TypeScript

React

SaaS features

On-call experience

AI tooling in engineering

Collaboration

Tools

OpenTelemetry

Job description

Role overview

Build observability features that help customers understand and troubleshoot AI agents in production. This full-stack role spans the data foundation—from ingestion and modeling to product experiences that reveal how agents behave, what they cost, and where they fail. You’ll work with a small cross-functional team and ship capabilities directly to customers.

Responsibilities

• Develop AI observability features across the pipeline, from data ingestion through in-product visualizations.
• Design and evolve data models and infrastructure for observing agents and their surrounding systems.
• Build ways to analyze and surface signals such as token use, cost, errors, and unusual behavior.
• Partner with product and design to shape clear, useful workflows for customers.
• Help customers and colleagues learn effective ways to explore agent behavior across conversations and workloads.
• Use rapid feedback and experimentation to improve features, sharing lessons from both successful and unsuccessful approaches.

Requirements

• Significant software engineering experience, including ownership of features from idea through delivery.
• Experience shipping customer-facing SaaS product features.
• Full-stack capability, with backend experience in Go and/or Python and frontend experience in TypeScript and React.
• Strong collaboration with product managers and designers.
• Regular, hands-on use of AI tools in engineering work.
• Experience participating in an on-call rotation; the role joins the rotation after roughly 90 days.
• Comfort working through changing requirements and ambiguity in a fast-moving area.

Nice to have

• Experience building production features powered by LLMs or agents.
• Familiarity with observability tools such as OpenTelemetry.

Benefits and work setup

• Fully distributed work model with trust and autonomy.
• Salary range: CAD $250,000–$280,000, based on experience level.
• Equity, unlimited paid time off, home office or coworking and internet support, employee benefits coverage, up to 16 weeks of paid parental leave, and an annual development allowance.