Remote Data Processing Engineer, I/O & Performance

DataPelago, Inc.; Mountain View (CA); Hybrid; USD 150,000 - 190,000; Full time

Job summary

DataPelago, Inc. is seeking a Data Processing Engineer - I/O to advance read/write capabilities of the DATA Pelago engine.

You will shape interfaces with data sources and sinks, catalogs, and formats, and drive performance across large-scale data sets. Ideal candidates bring 7+ years in data processing or distributed systems, deep experience with Spark/Parquet/ORC/Iceberg, and strong C/C++ with Rust preference.

Qualifications

• 7+ years in data processing or distributed systems.
• 3+ years optimizing read/write interfaces for large-scale data.
• Experience with Spark, Parquet, ORC, Iceberg, and similar stacks.
• Strong benchmarking and performance-tuning skills.
• Proficient in C/C++ with Rust preferred; Linux environments.

Responsibilities

• Architect data I/O interfaces with sources, sinks, catalogs, and data formats.
• Design adapters, data representations, and caching for the engine.
• Develop, test, optimize, and maintain core engine components.
• Evaluate open formats and libraries (Arrow, Parquet, Iceberg) to extend capabilities.
• Collaborate with engineering and product management on the roadmaps.
• Promote best practices in design reviews, testing, CI/CD, and quality.

Skills

C/C++

Rust

Performance optimization

Distributed systems

Linux development

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Apache Parquet

Apache ORC

Apache Iceberg

Apache Spark

Linux

Job description

DataPelago, Inc. is seeking a Data Processing Engineer - I/O to advance read/write capabilities of the DATA Pelago engine.

You will shape interfaces with data sources and sinks, catalogs, and formats, and drive performance across large-scale data sets. Ideal candidates bring 7+ years in data processing or distributed systems, deep experience with Spark/Parquet/ORC/Iceberg, and strong C/C++ with Rust preference.