Senior Director of Data Processing Engine Orchestration

NetApp; San Jose (CA); Hybrid; USD 228,000 - 339,000; Full time

Benefits offered by this job

Health Insurance

Employee stock purchase plan

Restricted stock units (RSUs)

Paid time off

Job summary

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting‑edge GenAI preprocessing. We are building an innovative data processing engine that transforms how Spark, Flink, Ray, and others operate on large datasets.

Our DataPelago Nucleus and DPA‑S power production applications for renowned customers. The Opportunity: as senior development leader, you will drive planning, optimization, and coordination of complex workloads on distributed systems, grow

Qualifications

• Bachelor’s degree in computer science or a related field.
• 12+ years of production software development experience with 7+ years of experience managing software development teams.
• 10+ years of experience in planning, developing, launching, and supporting core components of massively parallel data processing platforms for production deployment.
• Experience with OSS platforms such as Spark, Flink, Gluten, Velox, DataFusion, Spark RAPIDS.
• Strong leadership, talent development, and customer-facing skills.

Responsibilities

• Lead and grow a team of engineers focused on query planning and orchestration for massively parallel data processing engines.
• Drive development of industry-leading capabilities in data processing engine orchestration.
• Establish and improve metrics for product quality, performance, scale, and reliability in deployments.
• Improve SDLC using AI for higher code quality, release velocity, and productivity.
• Align product roadmap with organizational goals and priorities.
• Foster a high-performance, collaborative work culture and lead by example.

Skills

Team leadership

Query planning

Query optimization

Distributed systems

Data processing engines

People management

Education

Bachelor's degree in computer science

Master's or PhD preferred

Tools

Apache Spark

Apache Flink

Velox

Apache Gluten

DataFusion

Spark RAPIDS

Job description

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting‑edge GenAI preprocessing. We are building an innovative data processing engine that transforms how Spark, Flink, Ray, and others operate on large datasets.

Our DataPelago Nucleus and DPA‑S power production applications for renowned customers. The Opportunity: as senior development leader, you will drive planning, optimization, and coordination of complex workloads on distributed systems, grow