Senior Data Ingestion Engineer: Scalable Streaming

AI Chopping Block; San Francisco, Northern (CA, KY); Hybrid; USD 166,000 - 225,000; Full time

Job summary

Databricks is building the Ingestion Core Team to enable seamless, reliable ingestion of petabyte-scale data from diverse sources into Delta Lake. You will design distributed systems, optimize streaming workloads, and collaborate with teams to enable AI agents and RAG use cases.

The ideal engineer has 5+ years of production code experience in Java/Scala/Go/C++/Python and a strong background in distributed systems, Spark, and data processing.

Qualifications

• 5+ years of production code experience in Java, Scala, Go, C++, or Python.
• Experience architecting and deploying large-scale distributed/asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.

Responsibilities

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion and replication.
• Reduce end-to-end latency, increase throughput, and reduce costs from source to Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, latency, reliability, and scale.

Skills

Java

Scala

Go

C++

Python

Tools

Spark

Delta Lake

CDC

Job description

Databricks is building the Ingestion Core Team to enable seamless, reliable ingestion of petabyte-scale data from diverse sources into Delta Lake. You will design distributed systems, optimize streaming workloads, and collaborate with teams to enable AI agents and RAG use cases.

The ideal engineer has 5+ years of production code experience in Java/Scala/Go/C++/Python and a strong background in distributed systems, Spark, and data processing.