Senior Data Processing Engineer, OSS Engine & Spark/Flink

NetApp; United States; On-site; USD 196,000 - 293,000; Full time

Benefits offered by this job

Health Insurance

Retirement plan

Paid Time Off

Employee stock purchase plan

Job summary

DataPelago is building an innovative data processing engine to accelerate diverse data workloads on Spark, Flink and related OSS. We seek a Senior/Principal Data Processing Engineer - OSS to advance open-source integration, improve performance, and contribute to upstream and downstream projects.

You will influence architecture, design key enhancements, and collaborate across engineering, product, and community teams to push the frontier of accelerated data processing.

Qualifications

• 12+ years of relevant experience in data processing engines.
• 3+ years developing core components of Spark, Flink, or similar platforms.
• Experience instrumenting and optimizing performance on large workloads.
• Strong C/C++ and Java programming skills.
• Extensive Linux development experience.

Responsibilities

• Architect the interface between our engine and OSS platforms.
• Lead design of functional and performance enhancements for OSS integrations.
• Individually design, implement, test, and optimize engine components.
• Analyze OSS roadmaps to identify opportunities for leadership.
• Collaborate with engineering, product, and open-source communities.
• Promote code quality, testing, CI/CD, and issue resolution.

Skills

12+ years experience

Spark/Flink experience

Performance optimization

C/C++ and Java

Linux development

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Apache Spark

Apache Flink

Apache Gluten

Velox

Apache DataFusion

Presto/Trino

Doris

Job description

DataPelago is building an innovative data processing engine to accelerate diverse data workloads on Spark, Flink and related OSS. We seek a Senior/Principal Data Processing Engineer - OSS to advance open-source integration, improve performance, and contribute to upstream and downstream projects.

You will influence architecture, design key enhancements, and collaborate across engineering, product, and community teams to push the frontier of accelerated data processing.