Data Systems Engineer: Query Engines & Optimized Plans

Lever, Inc.; New York (NY); On-site; USD 135,000 - 200,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Relocation assistance

PTO

Job summary

Palantir Technologies in the United States seeks a software engineer to help build data processing infrastructure for Foundry, focusing on query planning, execution, and scalable storage. You will work with Spark, DataFusion, Iceberg, and Arrow, extending engines and tables to support diverse products.

You will own design-to-production improvements, collaborate with product teams, and ensure correctness and performance.

Qualifications

• 4+ years of professional software engineering experience building and operating production systems.
• Strong coding skills in Java, Rust, Scala, or C++.
• Experience with distributed data processing or database engines.

Responsibilities

• Designing and implementing query planning and optimization capabilities.
• Extending execution engines with new capabilities and improving operators, parallelism, memory management, and data movement.
• Developing Foundry’s Iceberg catalog and engine integrations, including table metadata and transactions.
• Building shared transformation semantics and execution interfaces for workloads across Foundry’s products.
• Improving incremental processing so pipelines can reuse previous results and process new data efficiently.

Skills

Java

Rust

Scala

C++

Education

Bachelor's degree in CS/Math/Engineering

Tools

Apache Spark

DataFusion

Iceberg

Arrow & Parquet

Job description

Palantir Technologies in the United States seeks a software engineer to help build data processing infrastructure for Foundry, focusing on query planning, execution, and scalable storage. You will work with Spark, DataFusion, Iceberg, and Arrow, extending engines and tables to support diverse products.

You will own design-to-production improvements, collaborate with product teams, and ensure correctness and performance.