Principal Data Processing Engineer (OSS Innovator)

NetApp; United States; On-site; USD 228,000 - 339,000; Full time

Job summary

DataPelago is seeking a Senior/Principal Data Processing Engineer - OSS to advance its data processing engine with OSS platforms like Apache Gluten, Velox, Spark and Flink. You will design and implement high-performance components, contribute upstream and downstream, and influence architecture for scalable processing.

This role offers opportunity to collaborate with a global OSS community, optimize performance for large-scale workloads, and shape the next generation of data acceleration

Qualifications

• 15+ years of relevant experience in data processing engineering.
• 3+ years building core components of large-scale data platforms.
• Proven ability to optimize performance of data processing engines.
• Strong knowledge of Spark, Flink, Presto/Trino or equivalent.
• Release-ready design, development and deployment track record.

Responsibilities

• Architect integration with OSS platforms and engines.
• Design and implement high-performance components of the engine.
• Lead performance enhancements and reliability improvements.
• Collaborate with OSS community and customer teams.
• Contribute upstream/downstream to selected OSS projects.
• Promote design reviews, CI/CD and code quality practices.

Skills

C/C++

Java

Linux

Open-source platforms

Performance optimization

Spark/Flink/Gluten/Velox experience

Education

Bachelor's degree in Computer Science or related field

Master's degree in Computer Science or related field

Tools

Apache Spark

Apache Flink

Velox

Gluten

DataFusion

Job description

DataPelago is seeking a Senior/Principal Data Processing Engineer - OSS to advance its data processing engine with OSS platforms like Apache Gluten, Velox, Spark and Flink. You will design and implement high-performance components, contribute upstream and downstream, and influence architecture for scalable processing.

This role offers opportunity to collaborate with a global OSS community, optimize performance for large-scale workloads, and shape the next generation of data acceleration