Platform Architect for Scalable Data Pipelines

Databricks; California (MO); On-site; USD 231,000 - 312,000; Full time

Job summary

Databricks is seeking a Senior Staff Engineer to serve as a technical architect and engineering leader for the Lakeflow Engineering platform. You'll own the long-term technical direction for declarative dataflow pipelines, materialized views, and streaming workloads powering thousands of production customers.

You will bridge research and production, lead multi-team initiatives, and drive deep systems work from performance diagnosis to low-level optimization, shaping the platform that underpins

Qualifications

• 10+ years of experience building, operating, and evolving large-scale distributed systems in production.
• Deep expertise in one or more: database internals, storage systems, distributed computing, streaming systems, language/API design, or performance engineering.
• Track record of executing against a multi-year technical vision through well-sequenced incremental milestones.
• Strong algorithmic foundations with practical judgment.
• Bias toward customer impact over technical novelty for its own sake.
• Experience aligning multiple teams and driving initiatives from conception through adoption.

Responsibilities

• Architect and deliver a highly scalable, fault-tolerant platform for declarative pipelines and materialized views serving thousands of production customers.
• Turn ambiguous, large-scale technical challenges into clear execution plans that ship incrementally and sustain at exabyte scale.
• Bridge the gap between research and production by applying ideas from literature or internal R&D to real customer problems.
• Lead deep systems work: performance diagnosis on large production clusters, low-level debugging, and optimization to reduce cost and latency.
• Shape product direction alongside engineering and product leadership, owning technical strategy from design to delivery.
• Drive complex, multi-team initiatives requiring sustained coordination and clear technical judgment.

Skills

Distributed systems

Performance engineering

Algorithms

Cross-team leadership

Customer impact

Education

BS, MS, or PhD in Computer Science or related field

Job description

Databricks is seeking a Senior Staff Engineer to serve as a technical architect and engineering leader for the Lakeflow Engineering platform. You'll own the long-term technical direction for declarative dataflow pipelines, materialized views, and streaming workloads powering thousands of production customers.

You will bridge research and production, lead multi-team initiatives, and drive deep systems work from performance diagnosis to low-level optimization, shaping the platform that underpins