Senior Data Systems Engineer - Open-Source, Distributed AI

LanceDB Inc.; United States; On-site; USD 180,000 - 260,000; Full time

Job summary

LanceDB Inc. is seeking a Senior Software Engineer to expand the reach of Lance and LanceDB within the data infrastructure ecosystem.

You will work at the intersection of high-performance computing, big data, and open-source systems to drive scale, performance, and maintainability. You will contribute to integrations with Spark, Trino, Hive Metastore, and Ray, implement efficient indexing and dataset operations, and help shape the Lance format across multiple data platforms.

Qualifications

• 10+ years of experience building high-performance databases, big data systems, or large-scale data services.
• Deep understanding of internals of open-source Big Data or AI training systems (e.g., Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX)
• Strong experience with high-performance computing in C++, Java, and/or Scala
• Experience with Rust (or willingness to learn it)
• Proven ability to move fast, work independently, and collaborate with a high-caliber team

Responsibilities

• Designing and maintaining efficient distributed Lance dataset operations
• Building efficient indices to enable predicate pushdown and accelerate queries in Spark, Ray, or Trino
• Working on table formats, data encodings, and various aspects of the Lance format in Rust
• Driving open-source community efforts to integrate the Lance format with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems
• Operating and improving internal data processing infrastructure
• Promoting the Lance format in open-source communities and at Big Data conferences

Skills

Open-source collaboration

High-performance computing

Distributed systems

Tools

Spark

Ray

Trino

Hadoop

Flink

Iceberg

Delta Lake

PyTorch

JAX

Apache Arrow

DataFusion

Parquet

Hive Metastore

Job description

LanceDB Inc. is seeking a Senior Software Engineer to expand the reach of Lance and LanceDB within the data infrastructure ecosystem.

You will work at the intersection of high-performance computing, big data, and open-source systems to drive scale, performance, and maintainability. You will contribute to integrations with Spark, Trino, Hive Metastore, and Ray, implement efficient indexing and dataset operations, and help shape the Lance format across multiple data platforms.