Senior Software Engineer – Distributed Data & Rust

Theory; Central (LA); On-site; USD 140,000 - 180,000; Full time

Job summary

LanceDB in the United States is seeking a Senior Software Engineer to extend Lance and LanceDB within the broader data infrastructure ecosystem. You will work at the intersection of high-performance computing, big data, and open-source systems, delivering scale and performance improvements, integrations with the wider data and AI ecosystem, and improvements in usability and maintainability.

You'll contribute to open-source community efforts, implement features in Rust and C++, and collaborate

Qualifications

• 10+ years of experience building high-performance databases, big data systems, or large-scale data services.
• Deep understanding of internals of open-source Big Data or AI training systems (e.g., Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX)
• Strong experience with high-performance computing in C++, Java, and/or Scala
• Experience with Rust (or willingness to learn it)
• Proven ability to move fast, work independently, and collaborate with a high-caliber team

Responsibilities

• Designing and maintaining efficient distributed Lance dataset operations
• Building efficient indices to enable predicate pushdown and accelerate queries in Spark, Ray, or Trino
• Working on table formats, data encodings, and various aspects of the Lance format in Rust
• Driving open-source community efforts to integrate the Lance format with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems
• Operating and improving internal data processing infrastructure
• Promoting the Lance format in open-source communities and at Big Data conferences

Skills

10+ years exp

Big Data/AI internals

C++/Java/Scala HPC

Rust learning

Independent collider

Job description

LanceDB in the United States is seeking a Senior Software Engineer to extend Lance and LanceDB within the broader data infrastructure ecosystem. You will work at the intersection of high-performance computing, big data, and open-source systems, delivering scale and performance improvements, integrations with the wider data and AI ecosystem, and improvements in usability and maintainability.

You'll contribute to open-source community efforts, implement features in Rust and C++, and collaborate