Data Engineering Manager

Burtch Works; Chicago (IL); On-site; USD 150,000 - 190,000; Full time

Job summary

Unknown Client Law Firm seeks a Data Engineering Manager to lead a 7–9 person team delivering an enterprise Databricks data platform. You will partner with Data Architecture and BI to ensure data transformations meet reporting needs while maintaining strong technical leadership.

The role blends technical execution with people management, offering a path to grow as a leader within a Data & AI organization. Hybrid work and enterprise-scale data initiatives are key aspects of this position.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science or a related field.
• Minimum of 5 years of hands-on data engineering experience, including scalable data pipelines and ETL/ELT processes.
• Minimum of 2 years of experience managing or leading a team of data engineers.
• Strong expertise with Databricks, Delta Lake, Databricks SQL, Apache Spark, Unity Catalog and Databricks Workflows, or comparable modern data platforms such as Snowflake.
• Proficiency with Python and SQL for large-scale data processing.
• Experience with Lakehouse architecture patterns and data modeling for analytics.
• Experience driving code reviews, setting engineering standards and data quality practices.
• Hands-on experience with cloud data platforms (Azure preferred).
• Leadership presence and strong communication across technical and business teams.

Responsibilities

• Lead a scrum team of data engineers, fostering technical excellence and collaboration.
• Conduct regular one-on-ones, performance reviews and career development conversations.
• Break engineering work into deliverables and drive the team against them.
• Set and enforce technical direction, standards and best practices across the Databricks platform.
• Lead technical design sessions translating business requirements into scalable solutions.
• Drive design and evolution of the Lakehouse architecture (Bronze/Silver/Gold) with Delta Lake, Spark and ADLS Gen2.
• Collaborate with Data Architect to align platform with data models and governance.
• Partner with BI team to satisfy downstream reporting transformations.
• Own and facilitate code reviews ensuring quality and maintainability.
• Establish data quality frameworks including validation, monitoring and SLA adherence.
• Oversee ETL/streaming pipelines using PySpark, Spark SQL, Delta Lake and Databricks Workflows.
• Develop reusable metadata-driven ingestion frameworks and modular data transformations.
• Manage consultants including performance management and conflict resolution.
• Troubleshoot platform and pipeline issues to ensure minimal downtime.

Skills

Leadership

Communication

Stakeholder management

Mentoring

Problem solving

Education

Bachelor's degree in CS/Engineering/Data Science or related field

Master's degree in CS/Engineering or related field

Tools

Databricks

Delta Lake

Databricks SQL

Apache Spark

Unity Catalog

Azure

Job description

Job Title: Data Engineering Manager

Location: Chicago, IL (hybrid - minimum three days onsite, two remote)

Employment Type: Permanent, full-time (exempt)

About The Company

Our client is a leading global law firm serving businesses, financial institutions and other organizations across a broad range of industries. Data & AI is a strategic priority for the firm, supported by a dedicated cross-functional organization of roughly 100 professionals spanning Data Engineering, AI Engineering, Product Management, UX, Data Governance, Business Intelligence and Program Management. The firm is actively investing in the development, evaluation, integration and responsible adoption of emerging AI technologies, and the team is building the next generation of enterprise data and AI capabilities used daily by lawyers and business professionals.

Job Summary

We are seeking a Data Engineering Manager to lead a scrum team of approximately 7-9 data engineers and consultants delivering our client's enterprise Databricks data platform. The platform blueprint and major architectural decisions have already been made, so this role is less about defining architecture from scratch and more about technical execution, data transformation, team leadership and delivery. The manager will partner closely with Data Architecture to understand how data should move through and be transformed within the platform, and with the Business Intelligence team to ensure the necessary transformations occur upstream of reporting and dashboards. Think of the role as roughly 60% technical and 40% leadership, while recognizing that it is ultimately a people-management position. This role reports to the Senior Manager of Data Platform & Engineering.

This is a strong opportunity for someone who wants to remain technically credible while becoming a stronger people leader, and who wants to join a Data & AI organization while major enterprise capabilities are still being built rather than after the transformation is complete.

Key Responsibilities

• Manage, mentor and develop a scrum team of data engineers, fostering a culture of technical excellence, collaboration and continuous improvement.
• Conduct regular one-on-ones, performance reviews and career development conversations to support individual growth and team retention.
• Break engineering work into manageable deliverables and drive the team against them; resolve impediments and shield engineers from organizational friction.
• Set and enforce technical direction, including coding standards, design patterns and engineering best practices across the Databricks data platform.
• Lead and participate in technical design sessions, translating complex business and data requirements into scalable, well-architected solutions.
• Drive the design and evolution of the Lakehouse architecture (Bronze/Silver/Gold), including Delta Lake, Apache Spark and ADLS Gen2.
• Collaborate with the Data Architect to align platform implementation with enterprise data models, domain definitions and governance standards.
• Partner with the Business Intelligence team to understand downstream reporting requirements and ensure the necessary transformations occur within the data platform.
• Own and facilitate the code review process, ensuring production code meets quality, performance and maintainability standards.
• Establish and enforce data quality frameworks, including validation, monitoring, alerting and SLA adherence across pipelines and data products.
• Oversee the end-to-end design, development and operation of scalable ETL and streaming pipelines using PySpark, Spark SQL, Delta Lake and Databricks Workflows.
• Drive the development of reusable, metadata-driven ingestion frameworks and modular data transformation patterns.
• Manage consultants, including performance management and conflict resolution.
• Troubleshoot and resolve complex platform, infrastructure and pipeline issues, ensuring minimal downtime and optimal performance.
• Help drive the platform toward production readiness and broader consumption as reporting capabilities come online.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science or a related field.
• Minimum of 5 years of hands‑on data engineering experience, including designing and building scalable data pipelines and ETL/ELT processes.
• Minimum of 2 years of experience managing or leading a team of data engineers, including direct people-management responsibilities.
• Strong expertise with Databricks, Delta Lake, Databricks SQL, Apache Spark, Unity Catalog and Databricks Workflows, or comparable modern data platforms such as Snowflake.
• Proficiency with Python and SQL for large-scale data processing.
• Proven experience with Lakehouse architecture patterns (Bronze/Silver/Gold), schema evolution and data modeling for analytics and operational workloads.
• Demonstrated experience driving code reviews, setting engineering standards and instilling data quality and testing disciplines within a team.
• Experience with CI/CD pipelines, version control, automated testing and monitoring in a data engineering context.
• Hands‑on experience with cloud data platforms in Azure, AWS or GCP, with Azure preferred.
• Sufficient recent technical depth to read and review code, assess quality, understand architecture, troubleshoot issues and provide credible technical direction. This role is not expected to write production code regularly.
• Leadership presence and the confidence to drive execution, hold engineers and consultants accountable, resolve conflict and manage performance.
• Strong communication and stakeholder management skills, with the ability to translate between technical and business contexts.

Preferred Qualifications

• Master's degree in Computer Science, Engineering or a related field.
• Experience integrating Databricks with Azure DevOps, ADLS Gen2 and Azure Key Vault.
• Familiarity with enterprise data modeling, data governance frameworks and metadata management tools such as Unity Catalog or Collibra.
• Experience with Infrastructure as Code (IaC) and Governance as Code practices.
• Familiarity with machine learning workloads and feature engineering in a Lakehouse environment.
• Experience leading data engineering teams in an agile or scrum delivery model.
• Industry experience in legal or professional services.

Other Skills And Abilities

• Strong organizational and project management skills.
• Strong attention to detail and commitment to quality.
• Good judgment and sound decision-making under pressure.
• Strong interpersonal and communication skills.
• Able to work harmoniously and effectively with others across technical and business teams.
• Able to preserve confidentiality and exercise discretion.
• Able to manage multiple priorities and competing deadlines.

Attendance

This is a full-time position. The role offers flexibility with a minimum of three days in the office per week. Regular, reliable attendance is expected and required. Additional hours may be required during periods of heavy workload, and flexibility in the daily work schedule is required to accommodate business requirements.

Applicants must be authorized to work in the United States without the need for employer sponsorship, now or in the future.