Senior Data Ingestion Platform Engineer

Cacheflow; San Francisco (CA); On-site; USD 166,000 - 225,000; Full time

Job summary

Databricks is seeking an experienced engineer for the Ingestion Core Team in San Francisco to help build distributed platform systems that ingest petabyte-scale data from diverse sources into Delta Lake with CDC and schema evolution.

You will reduce latency, increase throughput, and cut costs while designing streaming and batch ingestion workflows, monitoring, and observability. Collaboration with AI/Agent teams and SDKs will be part of the role.

Qualifications

• Production code experience in one of Java, Scala, Go, C++, or Python.
• Experience architecting large-scale distributed and asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
• Experience building or operating systems where scale, throughput, latency, reliability, and cost are important considerations.
• Comfortable using AI tools and defining effective evals to shorten the development loop.

Responsibilities

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication.
• Reduce end-to-end latency, increase throughput, and reduce costs from the time data appears in source systems to when it is available in Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
• Optimize streaming workloads by exploring and applying ML techniques.
• Build monitoring and observability capabilities (customer-facing and internal) that provide visibility into ingestion workflows and the systems running them.
• Collaborate with partner teams to enable use cases like RAG and AI agents.
• Build Agentic SDKs with strong evaluations.

Skills

5+ years coding (Java/Scala/Go/C++/Py)

Distributed systems

Streaming / Spark

High throughput / low latency

AI evals

Tools

Spark

Job description

Databricks is seeking an experienced engineer for the Ingestion Core Team in San Francisco to help build distributed platform systems that ingest petabyte-scale data from diverse sources into Delta Lake with CDC and schema evolution.

You will reduce latency, increase throughput, and cut costs while designing streaming and batch ingestion workflows, monitoring, and observability. Collaboration with AI/Agent teams and SDKs will be part of the role.