Databricks Data Platform Lead

LLP; Chicago (IL); Hybrid; USD 165,000 - 185,000; Full time

Job summary

Sidley Austin LLP seeks an Data Engineering Manager to lead a scrum team of data engineers responsible for the design, development, and delivery of the enterprise Databricks data platform. The role combines hands-on technical leadership with people management and sets technical standards across the Lakehouse architecture on Azure Databricks.

You will partner with Data Architect, analytics, and business teams to deliver scalable, governed data solutions and ensure quality, timeliness, and

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or related field.
• Minimum 5 years hands-on data engineering experience with scalable pipelines.
• At least 2 years leading a data engineering team with direct people management.
• Strong expertise in Azure Databricks and Lakehouse architectures.
• Proficiency in Python, PySpark, Spark SQL for large-scale data processing.
• Experience with CI/CD, version control, automated testing, and monitoring.
• Hands-on cloud experience (Azure preferred) with Databricks tools.
• Excellent written and verbal communication for stakeholder alignment.

Responsibilities

• Lead a scrum team of 5-7 data engineers, promoting technical excellence and collaboration.
• Conduct performance reviews and career development conversations.
• Remove impediments and shield the team to ensure delivery.
• Set engineering direction, coding standards, and best practices across Databricks.
• Translate business requirements into scalable, well-architected data solutions.
• Oversee end-to-end ETL/ELT and streaming pipelines on Azure Databricks.
• Drive reusable ingestion frameworks and data transformation patterns.
• Ensure data quality frameworks, monitoring, and SLA adherence.
• Manage code reviews and maintain production code quality.

Skills

Azure Databricks

PySpark

Spark SQL

Python

Delta Lake

Databricks Workflows

Unity Catalog

Code reviews

CI/CD pipelines

Stakeholder management

Education

Bachelor's degree in CS/Engineering/Data Science

5+ years data engineering

2+ years people leadership

Tools

Azure Databricks

Azure DevOps

Collibra

Delta Lake

Job description

Sidley Austin LLP seeks an Data Engineering Manager to lead a scrum team of data engineers responsible for the design, development, and delivery of the enterprise Databricks data platform. The role combines hands-on technical leadership with people management and sets technical standards across the Lakehouse architecture on Azure Databricks.

You will partner with Data Architect, analytics, and business teams to deliver scalable, governed data solutions and ensure quality, timeliness, and