Principal Engineer, AI Observability Platform (Remote)

HubSpot; Northern (KY); Hybrid; USD 314,000 - 502,000; Full time

Job summary

HubSpot is seeking a Principal Software Engineer to lead our Observability platform, spanning tracing, metrics, and logging across a vast microservices landscape. You will mentor engineers, set standards, and drive OpenTelemetry adoption while addressing AI observability challenges in production.

This role emphasizes hands-on coding, scalable architecture, and collaboration with diverse teams to ensure reliable, observable systems as HubSpot grows with AI-powered features.

Qualifications

• Platform-Builder experience in observability tooling for internal engineering teams.
• Telemetry pipeline design across high-cardinality data and high throughput.
• Incident automation and operational excellence with real-time telemetry workflows.

Responsibilities

• Define patterns for core telemetry platform (tracing, metrics, logging) at scale across hundreds of services.
• Lead strategy for tracing AI agents and ML-powered systems in production.
• Architect telemetry pipelines to balance sampling, fidelity, retention, and cost.

Skills

Platform-Builder

Telemetry Systems Depth

Incident Automation

Tools

OpenTelemetry

Distributed tracing engines

Metric backends

Log ingestion infrastructure

Job description

HubSpot is seeking a Principal Software Engineer to lead our Observability platform, spanning tracing, metrics, and logging across a vast microservices landscape. You will mentor engineers, set standards, and drive OpenTelemetry adoption while addressing AI observability challenges in production.

This role emphasizes hands-on coding, scalable architecture, and collaboration with diverse teams to ensure reliable, observable systems as HubSpot grows with AI-powered features.