Staff Software Engineer, Spark Structured Streaming Architect

Databricks Inc.; Seattle (WA); On-site; USD 182,400 - 247,000; Full time

Job summary

Databricks in Seattle is seeking an experienced Staff Engineer to drive the technical direction of Spark Structured Streaming, spanning OSS and Databricks components. Your mission is to make Spark Structured Streaming the state-of-the-art engine — adding advanced state management and new operators while re-imagining the engine architecture to improve latency, throughput, and cost.

You will set the technical vision, design core engine capabilities, mentor engineers, and collaborate with product

Qualifications

• BS or higher in Computer Science or related field, or equivalent practical experience.
• 8+ years building related systems — big‑data ecosystems, Apache Spark, or database internals.
• Comfortable working toward a multi‑year vision with incremental deliverables.
• Track record of delivering features with high operational quality and engineering discipline.
• Comfortable working cross‑functionally with product management and directly with customers.

Responsibilities

• Set and drive the technical vision for Spark Structured Streaming across OSS and Databricks Data Intelligence Platform.
• Design and build core engine capabilities — state management, new operators, and architectural improvements to latency and throughput.
• Raise the bar for engineering quality and operability, building software that is easy to run in production.
• Make company-wide impact by driving stream processing adoption across the Databricks product portfolio.
• Guide long-term architecture and technical‑debt decisions, balancing them against the product roadmap.
• Mentor and lead engineers on the team, and collaborate with the Engineering Manager to attract and grow top-tier talent.

Skills

Big-data systems

Apache Spark

Streaming

Software architecture

Education

BS in Computer Science or related field

Job description

Databricks in Seattle is seeking an experienced Staff Engineer to drive the technical direction of Spark Structured Streaming, spanning OSS and Databricks components. Your mission is to make Spark Structured Streaming the state-of-the-art engine — adding advanced state management and new operators while re-imagining the engine architecture to improve latency, throughput, and cost.

You will set the technical vision, design core engine capabilities, mentor engineers, and collaborate with product