Data Engineering Lead, Azure Databricks Lakehouse

Sidley Austin LLP; Chicago (IL); Hybrid; USD 165,000 - 185,000; Full time

Job summary

Sidley Austin LLP seeks a Data Engineering Manager to lead a scrum team delivering an Azure Databricks-based Lakehouse platform. You will balance hands-on technical leadership with people management, driving data quality, standards, and scalable pipelines across Bronze/Silver/Gold layers.

The role partners with analytics and business teams to set technical direction, review code, and ensure reliable, governable data solutions. Strong communication and collaboration are essential.

Qualifications

• 5+ years hands-on data engineering experience with scalable pipelines.
• 2+ years of people management or team leadership responsibilities.
• Strong Azure Databricks expertise including Lakehouse patterns and Unity Catalog.
• Proficiency with Python, PySpark, Spark SQL, and SQL for large-scale processing.
• Experience with CI/CD, version control, automated testing, and monitoring in data engineering.

Responsibilities

• Manage, mentor, and develop a scrum team of data engineers (5-7 members).
• Lead design sessions translating business requirements into scalable, well-architected solutions.
• Define and enforce technical direction, coding standards, and data quality frameworks.
• Oversee end-to-end ETL/ELT and streaming pipelines on Azure Databricks (Delta Lake).
• Drive reusable ingestion frameworks and metadata-driven data transformation patterns.
• Collaborate with Data Architect to align with enterprise data models and governance.

Skills

Azure Databricks

Python

PySpark

Spark SQL

SQL

Data pipelines

Data modeling

Leadership

Agile/Scrum

Code reviews

Education

Bachelor's degree in CS/Engineering/Data Science

Master's degree (preferred)

Tools

Delta Lake

Unity Catalog

Databricks Workflows

ADLS Gen2

Azure DevOps

Collibra

Job description

Sidley Austin LLP seeks a Data Engineering Manager to lead a scrum team delivering an Azure Databricks-based Lakehouse platform. You will balance hands-on technical leadership with people management, driving data quality, standards, and scalable pipelines across Bronze/Silver/Gold layers.

The role partners with analytics and business teams to set technical direction, review code, and ensure reliable, governable data solutions. Strong communication and collaboration are essential.