Data Platform Lead — Databricks & Lakehouse

Burtch Works; Chicago (IL); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Unknown Client Law Firm seeks a Data Engineering Manager to lead a 7–9 person team delivering an enterprise Databricks data platform. You will partner with Data Architecture and BI to ensure data transformations meet reporting needs while maintaining strong technical leadership.

The role blends technical execution with people management, offering a path to grow as a leader within a Data & AI organization. Hybrid work and enterprise-scale data initiatives are key aspects of this position.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science or a related field.
• Minimum of 5 years of hands-on data engineering experience, including scalable data pipelines and ETL/ELT processes.
• Minimum of 2 years of experience managing or leading a team of data engineers.
• Strong expertise with Databricks, Delta Lake, Databricks SQL, Apache Spark, Unity Catalog and Databricks Workflows, or comparable modern data platforms such as Snowflake.
• Proficiency with Python and SQL for large-scale data processing.
• Experience with Lakehouse architecture patterns and data modeling for analytics.
• Experience driving code reviews, setting engineering standards and data quality practices.
• Hands-on experience with cloud data platforms (Azure preferred).
• Leadership presence and strong communication across technical and business teams.

Responsibilities

• Lead a scrum team of data engineers, fostering technical excellence and collaboration.
• Conduct regular one-on-ones, performance reviews and career development conversations.
• Break engineering work into deliverables and drive the team against them.
• Set and enforce technical direction, standards and best practices across the Databricks platform.
• Lead technical design sessions translating business requirements into scalable solutions.
• Drive design and evolution of the Lakehouse architecture (Bronze/Silver/Gold) with Delta Lake, Spark and ADLS Gen2.
• Collaborate with Data Architect to align platform with data models and governance.
• Partner with BI team to satisfy downstream reporting transformations.
• Own and facilitate code reviews ensuring quality and maintainability.
• Establish data quality frameworks including validation, monitoring and SLA adherence.
• Oversee ETL/streaming pipelines using PySpark, Spark SQL, Delta Lake and Databricks Workflows.
• Develop reusable metadata-driven ingestion frameworks and modular data transformations.
• Manage consultants including performance management and conflict resolution.
• Troubleshoot platform and pipeline issues to ensure minimal downtime.

Skills

Leadership

Communication

Stakeholder management

Mentoring

Problem solving

Education

Bachelor's degree in CS/Engineering/Data Science or related field

Master's degree in CS/Engineering or related field

Tools

Databricks

Delta Lake

Databricks SQL

Apache Spark

Unity Catalog

Azure

Job description

Unknown Client Law Firm seeks a Data Engineering Manager to lead a 7–9 person team delivering an enterprise Databricks data platform. You will partner with Data Architecture and BI to ensure data transformations meet reporting needs while maintaining strong technical leadership.

The role blends technical execution with people management, offering a path to grow as a leader within a Data & AI organization. Hybrid work and enterprise-scale data initiatives are key aspects of this position.