Senior Software Engineer, Open-Source Data Systems (Rust)

LanceDB; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

LanceDB is seeking a Senior Software Engineer to expand Lance and LanceDB within the broader data infrastructure ecosystem. You’ll work at the intersection of high-performance computing, big data, and open-source systems, contributing scale and performance improvements, integrations with the wider data and AI ecosystem, and usability enhancements.

You will design and maintain distributed Lance dataset operations, build efficient indices for query acceleration across Spark, Ray, and Trino, work

Qualifications

• 10+ years building high-performance databases or data services.
• Strong knowledge of open-source big data training systems.
• Proficient in C++, Java, or Scala.
• Experience with Rust or willingness to learn.

Responsibilities

• Design and maintain distributed Lance dataset operations.
• Develop indices for predicate pushdown to accelerate queries.
• Contribute to Lance format in Rust and related table formats.
• Drive open-source integrations with Spark, Hive Metastore, Trino, Ray, and more.
• Promote Lance format in open-source communities and conferences.

Skills

C++/Java/Scala

Rust

Open-source involvement

Tools

Spark

Trino

Ray

Hadoop

Iceberg

Delta Lake

PyTorch

Job description

LanceDB is seeking a Senior Software Engineer to expand Lance and LanceDB within the broader data infrastructure ecosystem. You’ll work at the intersection of high-performance computing, big data, and open-source systems, contributing scale and performance improvements, integrations with the wider data and AI ecosystem, and usability enhancements.

You will design and maintain distributed Lance dataset operations, build efficient indices for query acceleration across Spark, Ray, and Trino, work