Azure Databricks Data Engineering Lead

5 Star Recruitment; Seattle (WA); On-site; USD 130,000 - 170,000; Full time

Job summary

A recruitment agency is seeking an experienced Azure Databricks Data Engineering Lead in Seattle, WA. The ideal candidate will have over 7 years of expertise in Databricks, Microsoft Azure, PySpark, and Python. Responsibilities include leading projects, building automation frameworks, and collaborating with cross-functional teams to optimize data management and governance. This role involves direct interaction with stakeholders to enhance data capabilities and drive business performance metrics. Excellent leadership and technical skills are required for this position.

Qualifications

• 7+ years of experience in Databricks, Microsoft Azure, PySpark and Python.
• Ability to lead large-scale, complex projects.
• Experience in building automation and monitoring frameworks.

Responsibilities

• Lead cross-functional projects to build technical roadmaps.
• Build infrastructure for optimal data extraction, transformation, and loading.
• Collaborate with business partners to identify opportunities.

Skills

Databricks

Microsoft Azure

PySpark

Python

Data pipeline management

Project leadership

Job description

About the job Azure Databricks Data Engineering Lead

Job Description

• 7+ years of experience in Databricks, Microsoft Azure, PySpark and Python.
• Lead large-scale, complex, cross-functional projects to build the technical roadmap for the WFM Data Services platform.
• Lead and review design artifacts.
• Build and own the automation and monitoring frameworks that showcase reliable, accurate, easy-to-understand metrics and operational KPIs to stakeholders for data pipeline quality.
• Execute proof of concept on new technology and tools to pick the best tools and solutions.
• Support business objectives by collaborating with business partners to identify opportunities and drive resolution.
• Communicate status and issues to Senior Starbucks leadership and stakeholders.
• Direct project team and cross-functional teams on all technical aspects of the projects.
• Lead with engineering team to build and support real-time, highly available data, data pipeline and technology capabilities.
• Translate strategic requirements into business requirements to ensure solutions meet business needs.
• Define and implement data retention policies and procedures.
• Define and implement data governance policies and procedures.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re‑designing infrastructure for greater scalability.
• Enable team to pursue insights and applied breakthroughs, while also driving the solutions to Starbucks scale.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of structured and unstructured data sources using big data technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data‑related technical issues and support their data infrastructure needs.
• Perform root cause analysis to identify permanent resolutions to software or business process issues.

MUST HAVE:

• 7+ years of experience in Databricks, Microsoft Azure, PySpark and Python.