Senior Data Ingestion Engineer - Scalable Streaming

Triwill Group; San Francisco, Northern (CA, KY); On-site; USD 166,000 - 225,000; Full time

Job summary

Databricks is seeking a Senior Software Engineer for the Ingestion Core team in San Francisco. You will help design and build distributed platform systems to ingest petabyte-scale data from cloud storage, databases, and file sources into Delta Lake with CDC and schema evolution.

You will reduce latency, improve throughput, and cut costs while collaborating with AI and data teams to enable RAG and AI agents, building monitoring, and delivering reliable streaming and batch ingestion at scale.

Qualifications

• 5+ years of production code experience in Java, Scala, Go, C++, or Python.
• Experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
• Experience building or operating systems where scale, throughput, latency, reliability, and cost are important considerations.
• Comfortable using AI tools and defining effective evals to shorten the development loop.

Responsibilities

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication.
• Reduce end-to-end latency, increase throughput, and reduce costs from the time data appears in source systems to when it is available in Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
• Optimize streaming workloads by exploring and applying ML techniques.
• Build monitoring and observability capabilities (customer-facing and internal) that provide visibility into ingestion workflows and the systems running them.
• Collaborate with partner teams to enable use cases like RAG and AI agents.
• Build Agentic SDKs with strong evaluations.

Skills

Java

Scala

Go

C++

Python

Distributed systems design

Streaming / Spark

CDC / data processing

AI tools familiarity

Job description

Databricks is seeking a Senior Software Engineer for the Ingestion Core team in San Francisco. You will help design and build distributed platform systems to ingest petabyte-scale data from cloud storage, databases, and file sources into Delta Lake with CDC and schema evolution.

You will reduce latency, improve throughput, and cut costs while collaborating with AI and data teams to enable RAG and AI agents, building monitoring, and delivering reliable streaming and batch ingestion at scale.