Senior Software Engineer, Distributed Data & Open-Source

LanceDB; United States; On-site; USD 150,000 - 190,000; Full time

Job summary

LanceDB is seeking a Senior Software Engineer to expand the Lance and LanceDB reach within the data infrastructure ecosystem. You will work at the intersection of high-performance computing, big data, and open-source systems, contributing scale, integration, and maintainability enhancements.

You will design distributed dataset operations, build indices for predicate pushdown, and advance the Lance format with Rust, Spark, and other major data frameworks, while collaborating with a top-tier team.

Qualifications

• 10+ years of experience building high-performance databases, big data systems, or large-scale data services.
• Deep understanding of internals of open-source big data or AI training systems (e.g., Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX)
• Strong experience with high-performance computing in C++, Java, and/or Scala

Responsibilities

• Designing and maintaining distributed Lance dataset operations
• Building efficient indices to accelerate queries in Spark, Ray, or Trino
• Working on table formats, data encodings, and Lance format in Rust
• Driving open-source community efforts to integrate Lance with Spark, Hive Metastore, Presto, Trino, Ray, and other data systems
• Operating and improving internal data processing infrastructure
• Promoting the Lance format in open-source communities and at Big Data conferences

Tools

Rust

C++

Java

Scala

Job description

LanceDB is seeking a Senior Software Engineer to expand the Lance and LanceDB reach within the data infrastructure ecosystem. You will work at the intersection of high-performance computing, big data, and open-source systems, contributing scale, integration, and maintainability enhancements.

You will design distributed dataset operations, build indices for predicate pushdown, and advance the Lance format with Rust, Spark, and other major data frameworks, while collaborating with a top-tier team.