Senior Data Infra Engineer (Rust, Spark, Open Source)

LanceDB Inc.; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

LanceDB Inc. is seeking a Senior Software Engineer to expand Lance and LanceDB within the data infrastructure ecosystem.

You will work at the intersection of HPC, big data, and open-source systems, delivering scale and performance improvements and enabling broader integrations. You will contribute to table formats, data encodings, and Rust-based Lance components while promoting the Lance format at conferences and in the OSS community.

Qualifications

• 10+ years of experience building high-performance data systems or databases.
• Strong background in distributed systems and open-source software.
• Experience with Spark, Trino, or similar data tooling.

Responsibilities

• Design and maintain distributed data operations for LanceDB.
• Build efficient indices to accelerate queries and predictor pushdown.
• Contribute to Lance formats, encodings, and Rust implementations.
• Drive integration efforts with Spark, Hive Metastore, Trino, Ray, and other systems.
• Improve internal data processing infrastructure and open-source collaboration.

Skills

Big data systems

C++/Java/Scala

Rust (learning)

Independent contributor

Tools

Hadoop

Spark

Iceberg

Trino/Presto

PyTorch

Job description

LanceDB Inc. is seeking a Senior Software Engineer to expand Lance and LanceDB within the data infrastructure ecosystem.

You will work at the intersection of HPC, big data, and open-source systems, delivering scale and performance improvements and enabling broader integrations. You will contribute to table formats, data encodings, and Rust-based Lance components while promoting the Lance format at conferences and in the OSS community.