Staff Software Engineer - Cross-Region Resilient Pipelines

Databricks; Mountain View (CA); On-site; USD 192,000 - 260,000; Full time

Job summary

Databricks seeks a Staff Engineer to join the Lakeflow Disaster Recovery team in Mountain View, CA. You will design and implement distributed systems that replicate and recover pipelines across regions, ensuring no data loss and correct ordering.

Key areas include cross-region replication, consistency guarantees, failover/failback workflows, and observability. You will work on high-availability streaming and batch ETL pipelines powering critical data products.

Qualifications

• 8+ years of experience working on distributed systems, databases, or storage.
• Strong software engineering skills in multiple production languages (Java/Scala/C++/Go/Python).
• Understanding of consistency, transactions, idempotency, replication, checkpointing, and data lineage.

Responsibilities

• Design and implement distributed systems that replicate and recover pipelines across regions.
• Develop cross-region replication for Lakeflow pipelines, streaming tables, and materialized views.
• Create failover and failback workflows with safety guardrails.
• Improve observability, testing, and failure-injection capabilities.
• Analyze and reason about failures with formal methods and simulations.

Skills

Java

Scala

C++

Go

Python

Education

PhD or advanced research in databases/distributed systems/storage

Job description

Databricks seeks a Staff Engineer to join the Lakeflow Disaster Recovery team in Mountain View, CA. You will design and implement distributed systems that replicate and recover pipelines across regions, ensuring no data loss and correct ordering.

Key areas include cross-region replication, consistency guarantees, failover/failback workflows, and observability. You will work on high-availability streaming and batch ETL pipelines powering critical data products.