Senior Software Engineer II - LLM Observability

honeycomb; United States; Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Equity

Unlimited PTO

Home-office support

Health coverage

Retirement contributions

Job summary

Honeycomb is seeking a senior backend engineer to build telemetry and observability foundations for production AI workloads. You will design APIs, implement services in Go, and deliver full-stack features with React/TypeScript interfaces.

This role requires mentoring teammates, leading multi-team initiatives, and participating in on-call rotations to maintain reliability across backend and frontend components. Fully distributed team with equity and generous benefits.

Qualifications

• Significant experience designing, building, and maintaining backend systems and APIs.
• Strong Go expertise including API design, error handling, maintainability, and developer experience.
• Ability to contribute across the stack with React/TypeScript and backend development.
• Experience leading technically complex, multi-team initiatives and making thoughtful tradeoffs around scale, cost, risk, and compatibility.

Responsibilities

• Design and implement maintainable, backward-compatible APIs and backend services, primarily in Go.
• Build full-stack product features using backend services and React/TypeScript interfaces.
• Shape technical patterns, operational guidance, and architecture for complex projects involving multiple teams or external dependencies.
• Work with product and design partners from discovery through delivery, using customer feedback to refine solutions and roadmaps.
• Operate and support software in production, including participation in an on-call rotation and proactive resolution of user pain points.
• Mentor engineers, share knowledge, communicate clearly, and delegate work to increase team effectiveness.

Skills

Mentorship

Cross-functional collaboration

On-call readiness

Technical leadership

Education

Bachelor's in CS or related field

Tools

Go

React/TypeScript

API design

Observability tooling

Job description

Role overview

Build the foundational systems that make production AI workloads observable and understandable. This senior engineering role focuses on representing traces, spans, token usage, costs, and evaluation signals in a consistent telemetry model, while delivering customer-facing observability capabilities across backend and frontend systems.

Responsibilities

• Design and implement maintainable, backward-compatible APIs and backend services, primarily in Go.
• Build full-stack product features using backend services and React/TypeScript interfaces.
• Shape technical patterns, operational guidance, and architecture for complex projects involving multiple teams or external dependencies.
• Work with product and design partners from discovery through delivery, using customer feedback to refine solutions and roadmaps.
• Operate and support software in production, including participation in an on-call rotation and proactive resolution of user pain points.
• Mentor engineers, share knowledge, communicate clearly, and delegate work to increase team effectiveness.

Requirements

• Significant experience designing, building, and maintaining backend systems, APIs, and production services.
• Strong expertise in Go or a comparable backend language, including API design, error handling, maintainability, and developer experience.
• Ability to contribute effectively across the stack with React/TypeScript and backend development.
• Experience leading technically complex, multi-team initiatives and making thoughtful tradeoffs around scale, cost, risk, and compatibility.
• Strong collaboration skills in a product-development partnership with design and product colleagues.
• Willingness to participate in on-call support and take ownership of software after release.
• Experience mentoring engineers and providing technical leadership through communication and feedback.

Nice to have

• Experience with AI or LLM systems, telemetry, observability, OpenTelemetry, or data models for production workloads.
• Familiarity with ecosystems such as agent frameworks, LLM tracing, evaluation platforms, or related developer tooling.

Benefits and work setup

• Fully distributed work with autonomy, trust, and accountability.
• Employee equity, transparent compensation levels, unlimited paid time off, and support for home-office or coworking needs.
• Employer-supported health coverage, retirement contributions, parental leave, and professional-development resources.