Senior Software Engineer – AI Agent Observability (Remote)

OpenKrill; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Generous equity and stock program

Unlimited PTO

Home office stipend

Full benefits for employees and depend

Parental leave up to 16 weeks

Development allowance

Job summary

Honeycomb is a fully distributed SaaS company building AI observability to help customers see what their agents are doing, their costs, and where errors happen. You’ll join a small, cross-functional team with engineers, a product manager, and a designer to ship features used in production.

You’ll work on end-to-end AI observability, collaborate with PMs and designers, and contribute to data models and infrastructure, with on-call rotation after about 90 days in a fast-moving environment that

Qualifications

• 3+ years of software engineering experience.
• Experience shipping customer-facing features at a SaaS company.
• Full-stack range, backend in Go and/or Python, frontend in TypeScript and React.
• Track record of working with product managers and designers.
• Regular, hands-on use of AI tools in day-to-day engineering work.
• Experience being on call (rotation after ~90 days).
• Ability to navigate ambiguity in a fast-moving field with shifting requirements and experimentation.
• Bonus: Experience building LLM or agent-powered features in production.
• Bonus: Hands-on experience with observability tooling, such as OpenTelemetry.

Responsibilities

• Deliver AI observability capabilities end to end, from ingestion to in-product visualizations.
• Contribute to the data model and infrastructure that observe agents and their ecosystems across the Honeycomb product.
• Build systems that analyze and surface AI-specific signals, such as cost, token usage, errors, and anomalies.
• Collaborate with product and design partners to ensure usability and unlock meaningful user workflows.
• Share learnings about exploring and understanding AI system behavior across conversations and workloads.
• Iterate through fast feedback loops and experimentation, sharing wins and failures.

Skills

Software engineering

SaaS experience

Full-stack Go/Python

TypeScript/React

Product collaboration

AI tools daily

On-call experience

Ambiguity navigation

LLM features

OpenTelemetry

Tools

OpenTelemetry

Job description

Honeycomb is a fully distributed SaaS company building AI observability to help customers see what their agents are doing, their costs, and where errors happen. You’ll join a small, cross-functional team with engineers, a product manager, and a designer to ship features used in production.

You’ll work on end-to-end AI observability, collaborate with PMs and designers, and contribute to data models and infrastructure, with on-call rotation after about 90 days in a fast-moving environment that