Data Infrastructure Intern: Fast Columnar Formats

Plus 2; Santa Clara (CA); On-site; USD 120,000 - 170,000; Full time

Job summary

PlusAI is seeking engineers to design and implement a fast columnar file format for autonomous vehicle data. You will optimize for fast topic reads, compression, and object storage compatibility, with the goal of scaling data processing via Spark.

The role emphasizes strong C++/Python programming, understanding of data structures and OS concepts, and familiarity with columnar formats like Parquet/Arrow/ORC. ROS2 compatibility may be pursued as time allows.

Qualifications

• Strong programming skills in C++ and Python.
• Solid understanding of data structures, algorithms, and OS concepts.
• Experience with large-scale data systems and data-intensive apps.
• Familiarity with columnar data formats and storage concepts.

Responsibilities

• Design and implement a fast columnar file format for storing data from autonomous vehicle operations.
• Benchmark formats for fast reads, compression, and object storage compatibility.
• Integrate format into existing data processing workflows (Spark or similar).
• Develop tooling to read/write the new format and enable ROS2 compatibility if possible.

Skills

C++

Python

Distributed data systems

Algorithms

Operating systems

File I/O performance

Tools

ROS2

Parquet

Arrow

ORC

Job description

PlusAI is seeking engineers to design and implement a fast columnar file format for autonomous vehicle data. You will optimize for fast topic reads, compression, and object storage compatibility, with the goal of scaling data processing via Spark.

The role emphasizes strong C++/Python programming, understanding of data structures and OS concepts, and familiarity with columnar formats like Parquet/Arrow/ORC. ROS2 compatibility may be pursued as time allows.