Staff Engineer, Spark Streaming Architect

Databricks; Seattle (WA); On-site; USD 182,400 - 247,000; Full time

Job summary

Databricks is seeking an experienced Staff Engineer to drive the technical direction of Spark Structured Streaming, spanning open source and Databricks components. You will define the architecture, shape latency/throughput improvements, and evolve a scalable streaming engine across platforms.

You will mentor engineers, collaborate with product teams, and help attract top talent while advancing a multi-year streaming vision in a dynamic data platform company.

Qualifications

• BS in Computer Science or related technical field.
• 8+ years building related systems such as big-data ecosystems, Apache Spark, or database internals.
• Strong interest in database systems, storage systems, distributed systems, language design, or performance optimization.
• Comfortable working toward a multi-year vision with incremental deliverables.
• Track record of delivering features with high engineering quality.
• Experience collaborating with product management and customers.

Responsibilities

• Set and drive the technical vision for Spark Structured Streaming across OSS and Databricks Data Intelligence Platform.
• Design and build core engine capabilities — state management, new operators, and architectural improvements to latency and throughput.
• Raise the bar for engineering quality and operability, building software that is easy to run in production.
• Make company-wide impact by driving stream processing adoption across the Databricks product portfolio.
• Guide long-term architecture and technical-debt decisions, balancing them against the product roadmap.
• Mentor and technically lead engineers on the team, and partner with the Engineering Manager to attract and grow top-tier talent.

Skills

8+ years exp

Database systems

Distributed systems

Performance optimization

Engineering quality

Cross-functional collaboration

Education

BS in CS

Job description

Databricks is seeking an experienced Staff Engineer to drive the technical direction of Spark Structured Streaming, spanning open source and Databricks components. You will define the architecture, shape latency/throughput improvements, and evolve a scalable streaming engine across platforms.

You will mentor engineers, collaborate with product teams, and help attract top talent while advancing a multi-year streaming vision in a dynamic data platform company.