Senior Principal Engineer, Real-Time Data Platform

International Business Machines Corporation; Armonk (NY); On-site; USD 250,000 - 360,000; Full time

Benefits offered by this job

Health benefits

Job summary

Confluent, a leader in real-time data streaming, seeks a Senior Principal Engineer to define the multi-year Background Plane strategy and drive architectural decisions across cloud storage, coordination engines, and SRE practices. You will mentor staff engineers and influence technology choices at scale.

You will shape end-to-end correctness, performance, and reliability, ensuring sub-millisecond foreground latencies while enabling massive planet-scale data streaming for enterprise customers.

Qualifications

• Proven track record architecting, building, and operating distributed systems at planetary scale.
• Deep knowledge of storage engines, distributed logs, stream processing, replication, or consensus protocols.
• Ability to articulate and execute multi-year architectural roadmaps across engineering orgs.
• Strong reasoning about race conditions, distributed concurrency, and backpressure.
• Experience establishing SLA/SLOs, durability targets, and automated recovery.
• Excellent written and verbal communication, able to align cross-team efforts.
• Mentor and sponsor senior engineers to grow leadership capacity.

Responsibilities

• Architect multi-year strategy for the Background Plane and align with business objectives.
• Design and optimize core platform capabilities: scheduling, coordination, isolation, autoscaling, backpressure, and cross-dependency workflows.
• Own correctness models for data and metadata workflows including compaction, rewriting, retention, and object GC under high concurrency.
• Ensure Background Plane scales with partition counts, throughput, storage, and latency targets.
• Lead architectural consensus across organizations on data references, metadata interfaces, and encryption models.
• Define and enforce global engineering standards for fault tolerance, chaos testing, observability, and incident response.
• Act as senior authority in architectural reviews, incident responses, and critical trade-off discussions.
• Sponsor and mentor engineers and represent Confluent’s technical vision externally.

Skills

Distributed systems

Streaming systems

Technical leadership

Problem solving

Reliability & SRE

Communication

Mentorship

Job description

Confluent, a leader in real-time data streaming, seeks a Senior Principal Engineer to define the multi-year Background Plane strategy and drive architectural decisions across cloud storage, coordination engines, and SRE practices. You will mentor staff engineers and influence technology choices at scale.

You will shape end-to-end correctness, performance, and reliability, ensuring sub-millisecond foreground latencies while enabling massive planet-scale data streaming for enterprise customers.