Confluent Staff Software Engineer - Kafka

International Business Machines Corporation; Boston (MA); Presencial; USD 180.000 - 250.000; Jornada completa

Descripción de la vacante

International Business Machines Corporation (IBM) is seeking a Staff Software Engineer to advance core layers of our distributed storage and query system. You will contribute to table materialization, background maintenance, and the analytics-driven query engine at scale.

You will work on multi-tenant, highly available compute infrastructure and collaborate with product and design teams to ensure seamless integration of storage and query capabilities with the broader platform.

Responsabilidades

• Storage Engine Development: Design and implement strategies for table schematization, materialization, and compaction at massive scale.
• Query Engine Development: Design and implement components for query planning, execution, and optimization for fast analytical queries.
• Platform Engineering: Build and maintain multi-tenant, highly available infrastructure powering background tasks for long-term storage and query serving.
• Cross-Functional Collaboration: Collaborate with product management, design, and other engineering teams to ensure seamless integration across features and infrastructure.

Descripción del empleo

About the Role

We are a team of passionate engineers who love solving complex distributed storage and query engine problems. Through the Tableflow and Lightning Tables project, we are building the next generation of storage and query solutions for Kora to enable streaming data to be consumed both as a table and as a stream seamlessly, and queried with the performance of a purpose-built analytical engine. As a Staff Software Engineer on the team, you will focus on advancing key layers of our system, including multi-tenant, highly available compute infrastructure, the core components responsible for table materialization and background maintenance, and the query engine that serves analytical workloads over that data. You will tackle challenges such as developing strategies for efficient table materialization and maintenance at massive scale across all of Confluent Cloud, building a fast and scalable query engine, and ensuring high availability and scalability for background storage tasks and query serving.

What You Will Do

• Storage Engine Development: Design and implement innovative strategies for table schematization, materialization, and compaction at massive scale.


• Query Engine Development: Design and implement components for query planning, execution, and optimization for fast analytical queries.


• Platform Engineering: Build and maintain multi-tenant, highly available infrastructure that powers background computational tasks for Kora's long-term storage and query serving. Ensure that the platform is optimized for scalability, reliability, and resilience, in alignment with organizational objectives.


• Cross-Functional Collaboration: Collaborate with product management, design, and other engineering teams to ensure seamless integration of storage and query features and infrastructure with the broader organization.