(Confluent) Senior Principal Engineer, Kafka

IBM; United States; Remote; USD 180,000 - 250,000; Full time

Job summary

Confluent seeks a Senior Principal Engineer to set the multi-year strategy for the Background Plane, impacting Kafka clusters at scale. You will lead architecture across cloud storage, consensus layers, and control planes while mentoring senior engineers and partnering with executive leadership.

The role emphasizes ex-ecutive-level influence, cross-functional collaboration, and delivering high-availability, low-latency streaming infrastructure for enterprise customers.

Qualifications

• Proven track record architecting and operating mission-critical distributed systems.
• Deep expertise in storage engines, distributed logs, or consensus protocols.
• Ability to articulate and execute multi-year architectural roadmaps.
• Strong problem-solving in distributed concurrency and backpressure.
• Experience establishing SLA/SLOs and automated recovery across multi-tenant infra.
• Excellent communication to influence executive leadership and cross-team alignment.
• Passion for mentoring senior engineers and building leadership capabilities.

Responsibilities

• Define long-term technical strategy for the Background Plane and multi-year roadmaps.
• Design and optimize core platform capabilities: scheduling, coordination, isolation, autoscaling, backpressure.
• Own correctness model for data/workflows across compaction, rewriting, retention, and GC.
• Scale background processing to support high growth in partitions and throughput.
• Lead architectural consensus across orgs on data references, interfaces, and task schedulers.
• Enforce global engineering standards for fault tolerance, chaos testing, observability, and incident response.
• Serve as senior technical authority during architectural reviews and critical incidents.
• Mentor staff engineers and represent the technical vision externally.

Skills

Distributed systems

Streaming & storage

Strategic leadership

Rigorous problem-solving

SRE & reliability

Executive communication

Mentorship & talent development

Education

Bachelor's Degree

Master's Degree

Job description

Introduction

Confluent is pioneering the foundational platform for real-time data streaming. Founded by the original creators of Apache Kafka, we enable forward-thinking organizations to connect applications, systems, and data streams, turning isolated data into actionable real-time intelligence. Our global engineering teams architect highly available, secure, distributed systems that operate at massive planetary scale. We are seeking extraordinary technical leaders who are energized by solving complex distributed systems challenges and driven to create profound, measurable impact for thousands of enterprise customers worldwide.

Your role and responsibilities

The Next-Generation Kafka Background Plane is the critical engine ensuring Kafka clusters remain performant, resilient, cost-efficient, and correct at scale. Operating behind the scenes, it governs data and metadata compaction, intelligent log rewriting, retention policies, segment lifecycle management, asynchronous object garbage collection, and multi-tenant task orchestration. At our scale, minor inefficiencies ripple across global infrastructure; getting it right empowers seamless, zero-downtime, infinite-scale data streaming for the modern enterprise.

As a Senior Principal Engineer, you will define and drive the multi-year technical strategy for the Background Plane, shaping its pivotal role within the broader Next-Gen Kafka architecture. This is an individual contributor role with company-wide reach and executive-level scope. You will lead high-ambiguity, highly visible initiatives spanning cloud storage, foreground serving engines, consensus layers, cloud infrastructure, control planes, and site reliability engineering—aligning senior technical leaders on mission-critical architecture calls while mentoring and sponsoring the next generation of principal talent.

What You Will Do

• Architect Multi-Year Strategy: Establish the long-term technical vision for the Background Plane, translating strategic business and product objectives into world-class distributed architectures, multi-year engineering roadmaps, and high-impact investment priorities.
• Engineer High-Scale Processing Platforms: Design, build, and optimize core platform capabilities including distributed scheduling, task coordination, isolation guarantees, elastic autoscaling, sophisticated backpressure, and cross-dependency execution workflows.
• Guarantee End-to-End Correctness: Own the mathematical and operational correctness model for all data and metadata workflows (compaction, rewriting, retention, segment lifecycle, object GC), building bulletproof safety guarantees under extreme concurrency, partial network failures, and degraded dependencies.
• Scale Without Bounds: Ensure the Background Plane seamlessly accommodates hyper-growth in partition counts, throughput, storage footprint, and background lag without compromising sub-millisecond foreground latencies.
• Drive Cross-Organizational Consensus: Spearhead architectural contracts across organizations—spanning data references, metadata interfaces, object life cycles, encryption models, and task schedulers—leading initiatives from initial framing to global production rollout.
• Raise Operational Excellence: Define and enforce global engineering standards for fault tolerance, chaos testing, deep observability, canary safety, and incident mitigation, continually driving down systemic risk across reliability, cost, and velocity.
• Serve as Senior Authority: Act as the definitive technical authority during critical architectural reviews, high-severity incident responses, and complex technical trade-off discussions spanning product, infrastructure, and security.
• Elevate Engineering Culture: Sponsor and mentor senior and staff engineers across the organization while representing Confluent’s technical vision externally through keynotes, research publications, and open-source contributions.

Required education

Bachelor's Degree

Preferred education

Master's Degree

Required technical and professional expertise

• Distributed Systems Mastery: Proven track record of architecting, building, and operating mission-critical distributed systems at planetary scale, with deep expertise in storage engines, distributed log stores, stream processing, replication, or consensus protocols.
• Streaming & Storage Expertise: Deep insight into Apache Kafka internals (storage semantics, log compaction, tiered storage, segment indexing) or equivalent high-throughput streaming systems.
• Strategic Technical Leadership: Demonstrated capability to articulate, execute, and deliver complex multi-year architectural roadmaps across multi-disciplinary engineering organizations.
• Rigorous Problem-Solving: Exceptional fluency in reasoning about race conditions, distributed concurrency, state machine transitions, edge-case failures, and backpressure mechanisms.
• Quality & SRE Rigor: Proven ability to establish measurable SLA/SLO metrics, durability targets, and automated recovery practices across global multi-tenant infrastructure fleets.
• Influential Communication: Outstanding written and oral communication skills, with an ability to distill highly technical concepts for executive leadership and forge cross-team alignment through trust and expertise.
• Talent Multiplier: A passion for mentorship, talent development, and empowering staff/principal engineers to reach their full leadership potential.

Preferred technical and professional experience

• Log-structured storage formats, custom storage engines, cloud object stores, or large-scale data re-writing pipelines.
• Fault-tolerant object lifecycle strategies designed to prevent data leakage, orphaned resources, or premature deletion.
• Chaos engineering, production traffic replay platforms, deterministic simulation testing, and large-scale load generation.
• Multi-cloud, multi-region deployment topologies across AWS, GCP, and Azure.

ABOUT BUSINESS UNIT

IBM Software infuses core business operations with intelligence—from machine learning to generative AI to help make organizations more responsive, productive, and resilient. IBM Software helps clients put AI into action now to create real value with trust, speed, and confidence across digital labor, IT automation, application modernization, security, and sustainability. Critical to this is the ability to make use of all data, because AI is only as good as the data that fuels it. In most organizations data is spread across multiple clouds, on premises, in private datacenters, and at the edge. IBM’s AI and data platform scales and accelerates the impact of AI with trusted data, and provides leading capabilities to train, tune and deploy AI across business. IBM’s hybrid cloud platform is one of the most comprehensive and consistent approach to development, security, and operations across hybrid environments—a flexible foundation for leveraging data, wherever it resides, to extend AI deep into a business.

In a world where technology never stands still, we understand that, dedication to our clients success, innovation that matters, and trust and personal responsibility in all our relationships, lives in what we do as IBMers as we strive to be the catalyst that makes the world work better.

Being an IBMer means you’ll be able to learn and develop yourself and your career, you’ll be encouraged to be courageous and experiment everyday, all whilst having continuous trust and support in an environment where everyone can thrive whatever their personal or professional background.

Our IBMers are growth minded, always staying curious, open to feedback and learning new information and skills to constantly transform themselves and our company. They are trusted to provide on-going feedback to help other IBMers grow, as well as collaborate with colleagues keeping in mind a team focused approach to include different perspectives to drive exceptional outcomes for our customers. The courage our IBMers have to make critical decisions everyday is essential to IBM becoming the catalyst for progress, always embracing challenges with resources they have to hand, a can-do attitude and always striving for an outcome focused approach within everything that they do.

Are you ready to be an IBMer?

ABOUT IBM

IBM’s greatest invention is the IBMer. We believe that through the application of intelligence, reason and science, we can improve business, society and the human condition, bringing the power of an open hybrid cloud and AI strategy to life for our clients and partners around the world.

Restlessly reinventing since 1911, we are not only one of the largest corporate organizations in the world, we’re also one of the biggest technology and consulting employers, with many of the Fortune 500 companies relying on the IBM Cloud to run their business.

At IBM, we pride ourselves on being an early adopter of artificial intelligence, quantum computing and blockchain. Now it’s time for you to join us on our journey to being a responsible technology innovator and a force for good in the world.

IBM is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, neurodivergence, age, or other characteristics protected by the applicable law. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.

OTHER RELEVANT JOB DETAILS

IBM offers a competitive and comprehensive benefits program. Eligible employees may have access to:

• Healthcare benefits including medical & prescription drug coverage, dental, vision, and mental health & well being
• Financial programs such as 401(k), cash balance pension plan, the IBM Employee Stock Purchase Plan, financial counseling, life insurance, short & long- term disability coverage, and opportunities for performance based salary incentive programs
• Generous paid time off including 12 holidays, minimum 56 hours sick time, 120 hours vacation, 12 weeks parental bonding leave in accordance with IBM Policy, and other Paid Care Leave programs. IBM also offers paid family leave benefits to eligible employees where required by applicable law
• Training and educational resources on our personalized, AI-driven learning platform where IBMers can grow skills and obtain industry-recognized certifications to achieve their career goals
• Diverse and inclusive employee resource groups, giving & volunteer opportunities, and discounts on retail products, services & experiences

We consider qualified applicants with criminal histories, consistent with applicable law.

The compensation range and benefits for this position are based on a full-time schedule for a full calendar year. The salary will vary depending on your job-related skills, experience and location. Pay increment and frequency of pay will be in accordance with employment classification and applicable laws. For part time roles, your compensation and benefits will be adjusted to reflect your hours. Benefits may be pro-rated for those who start working during the calendar year.