Principal Data Processing Engineer-OSS

DataPelago; Hyderabad; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

DataPelago in Hyderabad, India, is seeking a Principal Data Processing Engineer (OSS) to advance its data processing engine by integrating with open-source platforms like Apache Gluten, Velox, Spark, and Flink.

You will influence architecture, design and implement high-performance components, and collaborate with the OSS community and cross-functional teams to drive product leadership and reliability.

Qualifications

• BS/MS in Computer Science (or related field) with 6+ years of relevant experience.
• 3+ years of deep technical experience in instrumenting, analyzing, and optimizing performance of data processing engine components.
• Strong knowledge of architecture and internal operation of Spark, Flink, Presto/Trino.

Responsibilities

• Influence the architecture of how the data processing engine interfaces with open-source platforms.
• Lead design of functional and performance enhancements to open-source platforms and their integration with our engine.
• Individually design, implement, test, optimize, and maintain engine components.
• Analyze roadmaps of Gluten, Velox, Spark, Flink to identify opportunities for leadership.

Skills

C

C++

Java

Linux

Performance optimization

System architecture

Collaboration

Open source

Education

BS/MS in Computer Science

Tools

Apache Spark

Apache Flink

Presto/Trino

Apache Gluten

Velox

Job description

Principal Data Processing Engineer - OSS
Hyderabad, IN

About DataPelago:

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray, and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for world-class engineers to join our team and shape the future of accelerated data processing.

The Role:

As a Principal Data Processing Engineer (OSS), you will be a key individual contributor in adopting and advancing the capabilities of open-source software (OSS) platforms such as Apache Gluten, Velox, Apache Spark, and Apache Flink in the context of DataPelago's data processing engine. You will enhance the functional breadth, performance, scale, and reliability of the DataPelago engine through downstream and upstream contributions. You will have the opportunity to engage with the community working on these platforms. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.

What You'll Do:

• Influence the architecture of how our data processing engine interfaces with open-source platforms and engines.
• Lead the design of functional and performance enhancements to open source platforms such as Apache Gluten and Velox, and their integration with our data processing engine.
• Individually design, implement, test, optimize, and maintain components of the data processing engine.
• Analyze the technology roadmap of Apache Gluten, Velox, and equivalent platforms and identify opportunities for our engine to enhance technology and product leadership.
• Collaboration: Partner with engineering, product management, the open-source community and customer success teams.
• Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain the highest product quality, security efficiency, and productivity.

What You'll Bring:

• BS/MS in Computer Science (or a related field) with 6+ years of relevant experience
• 3+ years of deep technical experience in instrumenting, analyzing, and optimizing the performance of data processing engine components on benchmark and customer workloads.
• Sound knowledge of the architecture and internal operation of one or more of Apache Spark, Apache Flink, Presto/Trino.
• Demonstrated experience in the design, development, and successful release of high-performance data processing engines for large production deployments.
• Exceptional programming skills in C, C++, and Java.
• Extensive development experience in Linux environments.
• Excellent communication and collaboration skills, with the ability to articulate complex technical concepts to both technical and non-technical audiences.
• Strong analytical and problem-solving skills with a passion for performance optimization.

Why Join DataPelago?

• Technical Leadership: Take a leadership role in shaping the architecture and development of how our core engine works with open source data processing platforms
• Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated computing and data processing.
• Significant Impact: Your contributions will directly impact the performance and scalability of our mission-critical platform.
• Mentorship and Growth: Mentor and guide other talented engineers while expanding your own technical expertise.
• Competitive compensation, stock options, comprehensive benefits package, and leadership development opportunities

Location Considerations:

We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations.