Data Processing Engineer - IO Engineering Mountain View, CA / Hyderabad, IN / Remote

DataPelago, Inc.; Mountain View (CA); Remote; USD 150,000 - 190,000; Full time

Job summary

DataPelago, Inc. is seeking a Data Processing Engineer - I/O to advance read/write capabilities of the DATA Pelago engine.

You will shape interfaces with data sources and sinks, catalogs, and formats, and drive performance across large-scale data sets. Ideal candidates bring 7+ years in data processing or distributed systems, deep experience with Spark/Parquet/ORC/Iceberg, and strong C/C++ with Rust preference.

Qualifications

• 7+ years in data processing or distributed systems.
• 3+ years optimizing read/write interfaces for large-scale data.
• Experience with Spark, Parquet, ORC, Iceberg, and similar stacks.
• Strong benchmarking and performance-tuning skills.
• Proficient in C/C++ with Rust preferred; Linux environments.

Responsibilities

• Architect data I/O interfaces with sources, sinks, catalogs, and data formats.
• Design adapters, data representations, and caching for the engine.
• Develop, test, optimize, and maintain core engine components.
• Evaluate open formats and libraries (Arrow, Parquet, Iceberg) to extend capabilities.
• Collaborate with engineering and product management on the roadmaps.
• Promote best practices in design reviews, testing, CI/CD, and quality.

Skills

C/C++

Rust

Performance optimization

Distributed systems

Linux development

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Apache Parquet

Apache ORC

Apache Iceberg

Apache Spark

Linux

Job description

Data Processing Engineer - I/O

Mountain View, CA / Hyderabad, IN / Remote

About DataPelago:

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Ourteam of engineers drive and adopt advances in hardware-accelerated computing, parallel processingof large-scale data, query optimization, distributed systems, compilers, machine learning, andcloud-native computing. We are looking for specialists to join our engineering team and shape thefuture of accelerated data processing.

The Opportunity:

As a Data Processing Engineer - I/O, you will be a key individual contributor in advancing data read and write capabilities of DataPelago’s data processing engine. You will enhance functional breadth, performance, scale, and reliability of the DataPelago engine in reading and writing largescale data of various data types from diverse data sources and data sinks. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team ofengineers.

What You'll Do:

• Architect: Influence the architecture of how our data processing engine interfaces with data
  sources and sinks, catalogs, data formats.
• Design: Lead design of functional and performance enhancements to adapters/connectors,
  data representations, data filtering, caching and more in our data processing engine.
• Core Development: Individually design, implement, test, optimize, and maintain components of the data processing engine.
• Innovation and Differentiation: Analyze technology roadmap of existing and emerging data
  formats and libraries, open table formats, catalog services, and more (e.g., Apache Arrow,
  Apache Parquet, Apache Iceberg) and identify opportunities for our engine to enhance technology and product leadership.
• Collaboration: Partner effectively with engineering and product management in defining and
  realizing the data I/O roadmap of our product..
• Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD,
  and issue resolution to maintain highest product quality, security efficiency, & productivity.

What You'll Bring:

• Bachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant experience.
• 3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache ORC, Apache Iceberg, Apache Spark, and similar technologies.
• Demonstrated experience in instrumenting, analyzing, and optimizing the performance of
  data processing engine components on benchmark and customer workloads.
• Demonstrated experience in the design, development, and successful release of high-performance data processing engine features for large production deployments.
• Good knowledge of the architecture of one or more of Apache Spark, Apache Flink, Presto/
  Trino.
  Exceptional programming skills in C, C++. Rust experience preferred.
  Extensive development experience in Linux environments.
  Strong analytical and problem-solving skills with a passion for performance optimization.

Location Considerations:

We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and atremote locations.

Why Join DataPelago?

• Technology Leadership: Shape the architecture and development of how our core engine
  works with advanced data store platforms.
• Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated
  computing and data processing.
• Significant Impact: Your contributions will directly impact the performance and scalability
  of our mission-critical platform.
• Growth: Expand your technical expertise and scope of responsibilities working with other
  talented engineers and with a growing product.
• Competitive compensation, stock options, comprehensive benefits package, leadership development opportunities.