Confluent Staff Software Engineer - Kafka

IBM; Tucson (AZ); On-site; USD 150,000 - 190,000; Full time

Job summary

IBM Software is driving the next generation of AI-powered, cloud-native storage and analytic capabilities. As part of the Kora project, you will advance the storage and query stack, enabling streaming data to be consumed as both a table and a stream, with a high-performance analytical engine.

You will tackle multi-tenant, scalable compute infrastructure and ensure reliability across Confluent Cloud components.

Qualifications

• BS, MS, or PhD in computer science or a related field, or equivalent work experience.
• Excellent communication and collaboration skills, with the ability to influence and guide stakeholders at all levels.
• Experience with multi-tenant, highly available compute infrastructure, and the core components for table materialization and background maintenance.

Responsibilities

• Design and implement storage engine features, including table schematization, materialization, and maintenance at scale.
• Build and optimize a fast, scalable query engine for analytical workloads.
• Develop platform infrastructure powering background tasks for long-term storage and query serving with high availability and reliability.
• Collaborate with product, design and other engineering teams to integrate storage and query features across the system.

Skills

Distributed storage

Query engine development

Multi-tenant infrastructure

Cross-functional collaboration

Education

Master's Degree

Tools

Apache Kafka

Apache Iceberg

Apache Flink

Apache Fluss

Presto/Trino

Spark SQL

DuckDB

ClickHouse

AWS / Azure / GCP

Job description

IntroductionAt IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.

IntroductionAt IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.

IntroductionAt IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.Your Role And ResponsibilitiesAbout the Role:We are a team of passionate engineers who love solving complex distributed storage and query engine problems. Through the Tableflow and Lightning Tables project, we are building the next generation of storage and query solutions for Kora to enable streaming data to be consumed both as a table and as a stream seamlessly, and queried with the performance of a purpose-built analytical engine. As a Staff Software Engineer on the team, you will focus on advancing key layers of our system, including multi-tenant, highly available compute infrastructure, the core components responsible for table materialization and background maintenance, and the query engine that serves analytical workloads over that data. You will tackle challenges such as developing strategies for efficient table materialization and maintenance at massive scale across all of Confluent Cloud, building a fast and scalable query engine, and ensuring high availability and scalability for background storage tasks and query serving.What You Will Do

• Storage Engine Development: Design and implement innovative strategies for table schematization, materialization, and compaction at massive scale.
• Query Engine Development: Design and implement components for query planning, execution, and optimization for fast analytical queries.
• Platform Engineering: Build and maintain multi-tenant, highly available infrastructure that powers background computational tasks for Kora's long-term storage and query serving. Ensure that the platform is optimized for scalability, reliability, and resilience, in alignment with organizational objectives.
• Cross-Functional Collaboration: Collaborate with product management, design, and other engineering teams to ensure seamless integration of storage and query features and infrastructure with the broader organization.

Preferred EducationMaster's DegreeRequired Technical And Professional ExpertiseWhat You Will Bring:

• BS, MS, or PhD in computer science or a related field, or equivalent work experience
• Excellent communication and collaboration skills, with the ability to influence and guide stakeholders at all levels.

Preferred Technical And Professional ExperienceWhat Gives You an Edge:

• Familiarity with or has experience contributing to the following open-source technologies: Apache Kafka, Apache Iceberg, Apache Flink, Apache Fluss
• Experience with query engines or distributed SQL systems (e.g. Presto/Trino, Spark SQL, DuckDB, ClickHouse)
• Experience/knowledge with public clouds (AWS, Azure or GCP)