Data Platform Lead: Azure Databricks & Lakehouse

Sidley Austin LLP; Chicago (IL); On-site; USD 165,000 - 185,000; Full time

Job summary

Sidley Austin LLP seeks an Data Engineering Manager to lead a scrum team of data engineers in building a scalable Databricks-based data platform. You will balance hands-on leadership with architectural direction, setting standards and ensuring data quality across pipelines.

You will collaborate with Data Architect and business teams to translate requirements into robust, governed data solutions on Azure. Strong communication and stakeholder management are essential.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.

Responsibilities

• Manage, mentor, and develop a scrum team of 5-7 data engineers, fostering technical excellence and collaboration.
• Conduct regular one‑on‑ones, performance reviews, and career development conversations.
• Shield engineers from organizational friction to maximize delivery.
• Set and enforce technical direction, coding standards, and engineering best practices across the Databricks data platform.
• Lead technical design sessions translating business/data requirements into scalable solutions.
• Drive Lakehouse architecture evolution (Bronze/Silver/Gold) on Azure Databricks using Delta Lake, Spark, and ADLS Gen2.
• Collaborate with Data Architect to align platform with enterprise data models and governance.
• Own code review process ensuring production code meets quality and performance standards.
• Establish data quality frameworks including validation, monitoring, and SLA adherence across pipelines.
• Oversee end-to-end design and operation of scalable ETL/streaming pipelines on Azure Databricks.
• Develop reusable, metadata-driven ingestion frameworks and modular data transformations.
• Troubleshoot platform/infrastructure/pipeline issues to minimize downtime.

Skills

Azure Databricks

Python

PySpark

SQL

Spark SQL

Data modeling

Data pipelines

Leadership

Stakeholder management

CI/CD

Cloud platforms

Education

Bachelor's degree in CS/Engineering/Data Science

Master's degree (preferred)

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Databricks Notebooks

Azure DevOps

Job description

Sidley Austin LLP seeks an Data Engineering Manager to lead a scrum team of data engineers in building a scalable Databricks-based data platform. You will balance hands-on leadership with architectural direction, setting standards and ensuring data quality across pipelines.

You will collaborate with Data Architect and business teams to translate requirements into robust, governed data solutions on Azure. Strong communication and stakeholder management are essential.