Senior Data Ingestion Engineer - Core Platform

Databricks; San Francisco (CA); On-site; USD 166,000 - 225,000; Full time

Benefits offered by this job

Annual bonus

Equity

Job summary

Databricks is seeking an engineer to join the Ingestion Core Team in San Francisco, focusing on high-volume data ingestion and streaming. You will build distributed infrastructure to ingest data from diverse sources, ensure low latency and cost, and land data into Delta Lake with CDC and schema evolution.

You will work on streaming workloads, monitoring, and collaboration with partner teams to enable AI agents and RAG use cases.

Qualifications

• 5+ years of production code experience in Java, Scala, Go, C++, or Python.
• Experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
• Experience building or operating systems where scale, throughput, latency, reliability, and cost are important considerations.
• Comfortable using AI tools and defining effective evals to shorten the development loop.

Responsibilities

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication.
• Reduce end-to-end latency, increase throughput, and reduce costs from data source to Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
• Optimize streaming workloads by applying ML techniques.
• Build monitoring and observability capabilities for ingestion workflows and systems.
• Collaborate with partner teams to enable use cases like RAG and AI agents.

Skills

Production code

Distributed systems

Streaming

Spark

CDC

Java/Scala/Go/C++/Python

AI evals

Tools

Spark

Delta Lake

Databricks tooling

Job description

Databricks is seeking an engineer to join the Ingestion Core Team in San Francisco, focusing on high-volume data ingestion and streaming. You will build distributed infrastructure to ingest data from diverse sources, ensure low latency and cost, and land data into Delta Lake with CDC and schema evolution.

You will work on streaming workloads, monitoring, and collaboration with partner teams to enable AI agents and RAG use cases.