Staff Engineer, Lakeflow Disaster Recovery & Replication

Cacheflow; San Francisco (CA); On-site; USD 192,000 - 260,000; Full time

Job summary

Databricks is seeking a Staff Engineer for Lakeflow Disaster Recovery to design and implement distributed systems that replicate and recover pipelines across regions. You will tackle consistency, idempotency, and safe recovery without silent data loss or duplication.

You will focus on cross-region replication, failover workflows, observability, and high-fidelity recovery simulations within a multi-year technical vision, leveraging Java/Scala/C++/Go/Python across complex dataflows.

Qualifications

• 8+ years of experience on distributed systems or storage infrastructure.
• Deep knowledge of consistency, transactions, idempotency and data lineage.
• Ability to define and pursue a multi-year technical vision with iterative results.

Responsibilities

• Design and implement distributed systems that replicate and recover pipelines across regions.
• Solve problems around consistency, failover, and safe recovery without data loss or duplication.
• Lead cross-region replication, failover/failback workflows and testing.
• Develop high-fidelity recovery simulations and improve observability.
• Collaborate on metadata reconciliation and dataflow reliability.

Skills

Distributed systems

Java

Scala

C++

Go

Python

Education

PhD or advanced research in databases or distributed systems

8+ years of relevant experience

Job description

Databricks is seeking a Staff Engineer for Lakeflow Disaster Recovery to design and implement distributed systems that replicate and recover pipelines across regions. You will tackle consistency, idempotency, and safe recovery without silent data loss or duplication.

You will focus on cross-region replication, failover workflows, observability, and high-fidelity recovery simulations within a multi-year technical vision, leveraging Java/Scala/C++/Go/Python across complex dataflows.