Engineering Lead – Spark Structured Streaming

Databricks; Washington; On-site; USD 181,000 - 254,000; Full time

Job summary

Databricks is seeking an Engineering Leader to drive Spark™ Structured Streaming development for both open source and Databricks Intelligence Platform. You will lead an engineering team, shape architecture to balance latency and throughput, and align with product strategy.

You will recruit top talent, establish robust testing and SLAs, and collaborate across product management and customer teams to deliver high-quality streaming engine capabilities.

Qualifications

• Experience leading engineering teams in streaming or distributed data systems.
• Strong background in testing, quality, and SLAs for a product.
• Ability to attract, hire, and coach engineers to meet high standards.

Responsibilities

• Lead a talented engineering team in Spark Structured Streaming across OSS and Databricks Data Intelligence Platform.
• Oversee sustained recruitment of top-tier talent and upskill team members.
• Implement robust processes to execute product vision, strategy, and roadmap.
• Build software that is high quality and easy to operate.
• Drive Stream Processing adoption across the Databricks product portfolio.
• Manage technical debt and make long-term architecture decisions.

Skills

Streaming systems

Distributed systems

Database internals

Team leadership

Hiring & mentoring

Cross-functional collaboration

Job description

Databricks is seeking an Engineering Leader to drive Spark™ Structured Streaming development for both open source and Databricks Intelligence Platform. You will lead an engineering team, shape architecture to balance latency and throughput, and align with product strategy.

You will recruit top talent, establish robust testing and SLAs, and collaborate across product management and customer teams to deliver high-quality streaming engine capabilities.