Data Engineering Manager

Randstad Digital Americas; Chicago (IL); On-site; USD 165,000 - 185,000; Full time

Job summary

Randstad Digital Americas in Chicago is seeking a Data Engineering Manager to lead a scrum team of 5-7 data engineers and consultants, delivering the enterprise Databricks data platform.

You will partner with Data Architecture and BI to ensure data is ingested, transformed, and prepared for downstream reporting and dashboards, while setting technical standards and driving data quality across pipelines.

Qualifications

• Bachelor's degree in a related field.
• 5+ years of hands-on data engineering experience designing and building scalable data pipelines.
• 2+ years of team leadership or management experience.
• Strong expertise with Azure Databricks, Delta Lake, or similar platforms.
• Proficiency in Python and SQL for large-scale data processing.
• Experience with Lakehouse architectures (Bronze/Silver/Gold).
• Experience with CI/CD, automated testing, and monitoring in data engineering.

Responsibilities

• Lead and mentor a scrum team of 5-7 data engineers.
• Conduct one-on-ones, performance reviews, and career development discussions.
• Resolve impediments and shield engineers from organizational friction.
• Set technical direction and coding standards for the Databricks platform.
• Lead design sessions translating business data needs into scalable solutions.
• Drive Lakehouse architecture evolution on Azure Databricks including Delta Lake and Spark.
• Oversee data quality frameworks, monitoring, and SLA adherence.
• Ensure end-to-end ETL/streaming pipelines are scalable and reliable.

Skills

Leadership

Azure Databricks

Python

SQL

Data pipelines

Data quality

Communication

Mentorship

Technical depth

Education

Bachelor's degree

Tools

Databricks

Delta Lake

Apache Spark

Unity Catalog

Databricks Workflows

ADLS Gen2

Azure DevOps

Job description

Job Summary

Role Overview The Data Engineering Manager will lead a team of approximately 7-9 data engineers/consultants supporting the continued delivery of Sidley's enterprise Databricks data platform. The platform architecture and core infrastructure are largely established. This role is therefore less about defining architecture from scratch and more about technical execution, data transformation, team leadership, and delivery. The manager will work closely with Data Architecture and BI to ensure data is appropriately ingested, transformed, and prepared for downstream reporting and dashboards.

What We Are Looking For

• Strong data engineering fundamentals: 5+ years of hands-on data engineering experience with pipelines, ETL/ELT, data transformation, and modern cloud data platforms.
• Recent technical depth: This person will not be expected to write production code regularly, but must still be close enough to the technology to read and review code, assess quality, understand architecture, troubleshoot issues, and provide credible technical direction.
• People leadership: Ideally 2-4 years of recent mentorship/management experience. Strong Lead-level candidates with meaningful people-management experience may also be considered.
• Leadership presence: We need someone assertive and confident enough to drive execution, hold engineers and consultants accountable, resolve conflict, manage performance, and keep the team moving. Communication skills and leadership presence will be important throughout the interview process.

Technology

Azure Databricks specifically is NOT required. Please do not screen out strong candidates simply because their experience is on another cloud or comparable data platform. Strong experience with Databricks, Snowflake, or similar modern data platforms is relevant. Candidates should understand modern data architecture and be comfortable with technologies/concepts such as Python, SQL, ETL/ELT, Lakehouse architecture, CI/CD, testing, and data quality. Azure experience is preferred, but strong AWS/GCP cloud candidates should absolutely be considered.

Job Details

Location: Chicago, Illinois

Job type: Permanent

Salary: $165,000 - 185,000 per year

Work hours: 9am to 4pm

Education required: Bachelors

Responsibilities

The Data Engineering Manager will lead a scrum team of data engineers in the design, development, and delivery of Sidley's enterprise Databricks data platform. This role blends hands-on technical leadership with people management, balancing day-to-day engineering execution with longer-term architectural direction. Partnering closely with the Data Architect, analytics, and business teams, the Data Engineering Manager will set technical standards, drive data quality, and ensure the team delivers scalable, reliable, and governed data solutions. This role reports to the Senior Manager of Data Platform & Engineering.

• Manage, mentor, and develop a scrum team of 5-7 data engineers, fostering a culture of technical excellence, collaboration, and continuous improvement.
• Conduct regular one-on-ones, performance reviews, and career development conversations to support individual growth and team retention.
• Resolve team impediments and shield engineers from organizational friction so they can focus on delivery.
• Set and enforce technical direction for the team, including coding standards, design patterns, and engineering best practices across the Databricks data platform.
• Lead and participate in technical design sessions, translating complex business and data requirements into scalable, well-architected solutions.
• Drive the design and evolution of the Lakehouse architecture (Bronze/Silver/Gold) on Azure Databricks, including Delta Lake, Apache Spark, and ADLS Gen2.
• Collaborate with the Data Architect to align platform implementation with enterprise data models, domain definitions, and governance standards.
• Own and facilitate the code review process, ensuring all production code meets quality, performance, and maintainability standards.
• Establish and enforce data quality frameworks, including validation, monitoring, alerting, and SLA adherence across pipelines and data products.
• Oversee the end-to-end design, development, and operation of scalable ETL and streaming data pipelines on Azure Databricks, leveraging PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Drive the development of reusable, metadata-driven ingestion frameworks and modular data transformation patterns.
• Troubleshoot and resolve complex platform, infrastructure, and pipeline issues, ensuring minimal downtime and optimal performance.

Education and/or Experience

Required

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
• A minimum of 5 years of hands-on experience in data engineering, including designing and building scalable data pipelines and ETL/ELT processes.
• A minimum of 2 years of experience managing or leading a team of data engineers, including direct people management responsibilities.
• Strong expertise in Azure Databricks, Delta Lake, Databricks SQL, Apache Spark, Unity Catalog, Databricks Workflows, or similar data platforms.
• Proficiency with Python and SQL for large-scale data processing.
• Proven experience with Lakehouse architecture patterns (Bronze/Silver/Gold), schema evolution, and data modeling for analytics and operational workloads.
• Demonstrated experience driving code reviews, setting engineering standards, and instilling data quality and testing disciplines within a team.
• Experience with CI/CD pipelines, version control, automated testing, and monitoring in a data engineering context.
• Hands-on experience with cloud data platforms in Azure, AWS, or GCP, with Azure strongly preferred.
• Strong communication and stakeholder management skills, with the ability to translate between technical and business contexts.

Preferred

• Master's degree in Computer Science, Engineering, or a related field.
• Experience integrating Azure Databricks with Azure DevOps, ADLS Gen2, and Azure Key Vault.
• Familiarity with enterprise data modeling, data governance frameworks, and metadata management tools such as Unity Catalog or Collibra.
• Experience with Infrastructure as Code (IaC) and Governance as Code practices.
• Familiarity with machine learning workloads and feature engineering in a Lakehouse environment.
• Experience leading data engineering teams in an agile or scrum delivery model.
• Industry experience in legal or professional services a plus.

Other Skills and Abilities

• Strong organizational and project management skills.
• Strong attention to detail and commitment to quality.
• Good judgment and sound decision-making under pressure.
• Strong interpersonal and communication skills.
• Able to work harmoniously and effectively with others across technical and business teams.
• Able to preserve confidentiality and exercise discretion.
• Able to manage multiple priorities and competing deadlines.

Skills

• AWS
• review code
• platform architecture
• CI/CD
• data analysis
• Data Architecture
• modern data architecture
• Lakehouse architecture
• data platform
• modern data platforms
• data platforms
• data transformation
• Databricks
• Azure Databricks
• ETL
• Data Engineering
• Azure
• production code
• Python
• requirements documentation
• SQL
• Snowflake
• assertive
• Communication skills
• confident
• People leadership
• Leadership
• credible
• architecture
• blueprint
• conflict resolution
• dashboards
• data quality
• infrastructure
• mentorshipperformance management
• production readiness
• project plans
• quality standards
• lead a team
• team leadership
• management experience
• technical depth
• technical direction
• testing

Equal Opportunity Employer

Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact [e-mail].

Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.