Data Engineering Manager

LLP; Chicago (IL); Hybrid; USD 165,000 - 185,000; Full time

Job summary

Sidley Austin LLP seeks an Data Engineering Manager to lead a scrum team of data engineers responsible for the design, development, and delivery of the enterprise Databricks data platform. The role combines hands-on technical leadership with people management and sets technical standards across the Lakehouse architecture on Azure Databricks.

You will partner with Data Architect, analytics, and business teams to deliver scalable, governed data solutions and ensure quality, timeliness, and

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or related field.
• Minimum 5 years hands-on data engineering experience with scalable pipelines.
• At least 2 years leading a data engineering team with direct people management.
• Strong expertise in Azure Databricks and Lakehouse architectures.
• Proficiency in Python, PySpark, Spark SQL for large-scale data processing.
• Experience with CI/CD, version control, automated testing, and monitoring.
• Hands-on cloud experience (Azure preferred) with Databricks tools.
• Excellent written and verbal communication for stakeholder alignment.

Responsibilities

• Lead a scrum team of 5-7 data engineers, promoting technical excellence and collaboration.
• Conduct performance reviews and career development conversations.
• Remove impediments and shield the team to ensure delivery.
• Set engineering direction, coding standards, and best practices across Databricks.
• Translate business requirements into scalable, well-architected data solutions.
• Oversee end-to-end ETL/ELT and streaming pipelines on Azure Databricks.
• Drive reusable ingestion frameworks and data transformation patterns.
• Ensure data quality frameworks, monitoring, and SLA adherence.
• Manage code reviews and maintain production code quality.

Skills

Azure Databricks

PySpark

Spark SQL

Python

Delta Lake

Databricks Workflows

Unity Catalog

Code reviews

CI/CD pipelines

Stakeholder management

Education

Bachelor's degree in CS/Engineering/Data Science

5+ years data engineering

2+ years people leadership

Tools

Azure Databricks

Azure DevOps

Collibra

Delta Lake

Job description

The Data Engineering Manager will lead a scrum team of data engineers in the design, development, and delivery of Sidley's enterprise Databricks data platform. This role blends hands-on technical leadership with people management, balancing day-to-day engineering execution with longer-term architectural direction. Partnering closely with the Data Architect, analytics, and business teams, the Data Engineering Manager will set technical standards, drive data quality, and ensure the team delivers scalable, reliable, and governed data solutions. This role reports to the Senior Manager of Data Platform & Engineering.

Duties and Responsibilities:

• Manage, mentor, and develop a scrum team of 5-7 data engineers, fostering a culture of technical excellence, collaboration, and continuous improvement.
• Conduct regular one-on-ones, performance reviews, and career development conversations to support individual growth and team retention.
• Resolve team impediments and shield engineers from organizational friction so they can focus on delivery.
• Set and enforce technical direction for the team, including coding standards, design patterns, and engineering best practices across the Databricks data platform.
• Lead and participate in technical design sessions, translating complex business and data requirements into scalable, well-architected solutions.
• Drive the design and evolution of the Lakehouse architecture (Bronze/Silver/Gold) on Azure Databricks, including Delta Lake, Apache Spark, and ADLS Gen2.
• Collaborate with the Data Architect to align platform implementation with enterprise data models, domain definitions, and governance standards.
• Own and facilitate the code review process, ensuring all production code meets quality, performance, and maintainability standards.
• Establish and enforce data quality frameworks, including validation, monitoring, alerting, and SLA adherence across pipelines and data products.
• Oversee the end-to-end design, development, and operation of scalable ETL and streaming data pipelines on Azure Databricks, leveraging PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Drive the development of reusable, metadata-driven ingestion frameworks and modular data transformation patterns.
• Troubleshoot and resolve complex platform, infrastructure, and pipeline issues, ensuring minimal downtime and optimal performance.

Education and/or Experience:

Required:

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
• A minimum of 5 years of hands-on experience in data engineering, including designing and building scalable data pipelines and ETL/ELT processes.
• A minimum of 2 years of experience managing or leading a team of data engineers, including direct people management responsibilities.
• Strong expertise in Azure Databricks, including Databricks Lakehouse, Delta Lake, Databricks SQL, Apache Spark, Unity Catalog, Databricks Workflows, and Databricks Notebooks.
• Proficiency with Python, PySpark, Spark SQL, and SQL for large-scale data processing.
• Proven experience with Lakehouse architecture patterns (Bronze/Silver/Gold), schema evolution, and data modeling for analytics and operational workloads.
• Demonstrated experience driving code reviews, setting engineering standards, and instilling data quality and testing disciplines within a team.
• Experience with CI/CD pipelines, version control, automated testing, and monitoring in a data engineering context.
• Hands-on experience with cloud data platforms in Azure, AWS, or GCP, with Azure strongly preferred.
• Strong communication and stakeholder management skills, with the ability to translate between technical and business contexts.

Preferred:

• Master's degree in Computer Science, Engineering, or a related field.
• Experience integrating Azure Databricks with Azure DevOps, ADLS Gen2, and Azure Key Vault.
• Familiarity with enterprise data modeling, data governance frameworks, and metadata management tools such as Unity Catalog or Collibra.
• Experience with Infrastructure as Code (IaC) and Governance as Code practices.
• Familiarity with machine learning workloads and feature engineering in a Lakehouse environment.
• Experience leading data engineering teams in an agile or scrum delivery model.
• Industry experience in legal or professional services a plus.

Other Skills and Abilities:

The following will also be required of the successful candidate:

• Strong organizational and project management skills.
• Strong attention to detail and commitment to quality.
• Good judgment and sound decision-making under pressure.
• Strong interpersonal and communication skills.
• Able to work harmoniously and effectively with others across technical and business teams.
• Able to preserve confidentiality and exercise discretion.
• Able to manage multiple priorities and competing deadlines

#LI-OE1

#LI-Hybrid

Applicants must be authorized to work in the United States without the need for employer sponsorship, now or in the future

The target salary range for this role is:

$165,000 - $185,000 if located in Illinois.

Salaries vary by location and are based on numerous factors, including, but not limited to, the relevant market, skills, experience, and education of the selected candidate. Our compensation package also includes bonus eligibility and a comprehensive benefits program. Benefits information can be found at Sidley.com/Benefits.

To perform this job successfully, an individual must be able to perform the Duties and Responsibilities above satisfactorily and meet the requirements. The requirements listed above are representative of the minimum knowledge, skill, and/or ability required. Reasonable accommodations will be made to enable individuals with disabilities to perform the essential functions of the job. If you need such an accommodation, please email [e-mail] (current employees should contact Human Resources).

Sidley Austin LLP is an Equal Opportunity Employer.