Senior Principal Engineer, Real-Time Data Platform

International Business Machines Corporation; Dallas (TX); On-site; USD 210,000 - 290,000; Full time

Job summary

Confluent is seeking a Senior Principal Engineer to define and drive the multi-year strategy for the Background Plane within the Next-Gen Kafka architecture. This individual contributor role operates with executive-level scope and company-wide reach.

You will lead high-ambiguity initiatives across cloud storage, consensus layers, and site reliability engineering, guiding senior leaders and mentoring staff engineers.

Qualifications

• Proven track record architecting, building, and operating mission-critical distributed systems at planetary scale.
• Deep insight into Apache Kafka internals or equivalent high-throughput streaming systems.
• Ability to articulate, execute, and deliver long-term architectural roadmaps across engineering organizations.
• Fluency in reasoning about race conditions, distributed concurrency, state machine transitions, edge-case failures, and backpressure.

Responsibilities

• Architect multi-year strategy translating business objectives into world-class distributed architectures and roadmaps.
• Design, build, and optimize core platform capabilities including scheduling, coordination, isolation guarantees, autoscaling, backpressure, and cross-dependency workflows.
• Own correctness models for data and metadata workflows (compaction, rewriting, retention, object GC) ensuring safety under failures and concurrency.
• Scale background processing to accommodate hyper-growth in partitions, throughput, storage, and latency.
• Drive architectural consensus across organizations spanning data references, interfaces, life cycles, and encryption models.
• Define and enforce global engineering standards for fault tolerance, chaos testing, observability, canary safety, and incident mitigation.

Skills

Distributed systems

Streaming & storage

Strategic technical leadership

Rigorous problem solving

Quality & SRE rigor

Influential communication

Talent multiplier

Tools

Multi-cloud platforms

Job description

Confluent is seeking a Senior Principal Engineer to define and drive the multi-year strategy for the Background Plane within the Next-Gen Kafka architecture. This individual contributor role operates with executive-level scope and company-wide reach.

You will lead high-ambiguity initiatives across cloud storage, consensus layers, and site reliability engineering, guiding senior leaders and mentoring staff engineers.