Staff Engineer, Spark Structured Streaming

Menlo Ventures; Seattle (WA); On-site; USD 182,400 - 247,000; Full time

Job summary

Databricks is seeking an experienced Staff Engineer to steer the technical direction of Spark Structured Streaming, spanning open source and Databricks components. The role focuses on state management, new operators, and architecture to improve latency, throughput, and cost.

You will mentor engineers, influence product roadmaps, and collaborate with customers to understand needs and deliver robust streaming capabilities across the platform.

Qualifications

• 8+ years building related systems—big-data ecosystems, Spark, or database internals.
• Passion for database/storage/distributed systems, language design, or optimization.
• Comfortable with multi-year vision and incremental delivery.

Responsibilities

• Set and drive the technical vision for Spark Structured Streaming across OSS and Databricks platform.
• Design and build core engine capabilities—state management, new operators, latency/throughput improvements.
• Raise the bar for engineering quality and operability for production-grade software.
• Drive stream processing adoption across the Databricks product portfolio.

Skills

Big-data systems

Apache Spark

Distributed systems

Language design

Performance optimization

Cross-functional collaboration

Education

BS or higher in Computer Science or related field

Job description

Databricks is seeking an experienced Staff Engineer to steer the technical direction of Spark Structured Streaming, spanning open source and Databricks components. The role focuses on state management, new operators, and architecture to improve latency, throughput, and cost.

You will mentor engineers, influence product roadmaps, and collaborate with customers to understand needs and deliver robust streaming capabilities across the platform.