Principal Data Processing Engineer, OSS & Spark

NetApp; San Jose (CA); Hybrid; USD 228,000 - 339,000; Full time

Benefits offered by this job

Health Insurance

Stock options (RSUs)

Retirement plan

Hybrid work model

Job summary

NetApp is seeking a Senior/Principal Data Processing Engineer - OSS to advance DataPelago’s engine by integrating with Apache Gluten, Velox, Spark, and Flink. You will contribute to architecture, design, and high-performance implementations for large-scale data processing.

You will collaborate with open-source communities, product teams, and customers while shaping the roadmap for acceleration across CPU/GPU platforms. Hybrid work and global collaboration are supported.

Qualifications

• 15+ years of relevant experience in data processing or related field.
• 3+ years designing core components in Spark, Flink, Doris, Gluten, Velox or similar.
• 3+ years optimizing performance of data processing engines.
• Deep knowledge of Spark/Flink and related architectures.
• Proven track record releasing high-performance engines for production deployments.

Responsibilities

• Architect data processing engine interfaces with OSS platforms.
• Lead design of enhancements for Gluten, Velox, Spark and Flink integrations.
• Individually design, implement, test, optimize, and maintain engine components.
• Analyze roadmaps to identify differentiation opportunities for the engine.
• Collaborate with engineering, product, OSS communities, and customer success teams.
• Promote best practices in design reviews, testing, CI/CD and reliability.

Skills

C/C++

Java

Linux

Spark/Flink

Open-source contributions

Performance optimization

Data processing engines

Education

Bachelor's or Master's in Computer Science

Tools

Apache Gluten

Velox

Apache Spark

Apache Flink

Apache DataFusion

DataFusion Comet

Job description

NetApp is seeking a Senior/Principal Data Processing Engineer - OSS to advance DataPelago’s engine by integrating with Apache Gluten, Velox, Spark, and Flink. You will contribute to architecture, design, and high-performance implementations for large-scale data processing.

You will collaborate with open-source communities, product teams, and customers while shaping the roadmap for acceleration across CPU/GPU platforms. Hybrid work and global collaboration are supported.