Data Platform Lead: Databricks, ETL & Cloud Delivery

Randstad Digital Americas; Chicago (IL); On-site; USD 165,000 - 185,000; Full time

Job summary

Randstad Digital Americas in Chicago is seeking a Data Engineering Manager to lead a scrum team of 5-7 data engineers and consultants, delivering the enterprise Databricks data platform.

You will partner with Data Architecture and BI to ensure data is ingested, transformed, and prepared for downstream reporting and dashboards, while setting technical standards and driving data quality across pipelines.

Qualifications

• Bachelor's degree in a related field.
• 5+ years of hands-on data engineering experience designing and building scalable data pipelines.
• 2+ years of team leadership or management experience.
• Strong expertise with Azure Databricks, Delta Lake, or similar platforms.
• Proficiency in Python and SQL for large-scale data processing.
• Experience with Lakehouse architectures (Bronze/Silver/Gold).
• Experience with CI/CD, automated testing, and monitoring in data engineering.

Responsibilities

• Lead and mentor a scrum team of 5-7 data engineers.
• Conduct one-on-ones, performance reviews, and career development discussions.
• Resolve impediments and shield engineers from organizational friction.
• Set technical direction and coding standards for the Databricks platform.
• Lead design sessions translating business data needs into scalable solutions.
• Drive Lakehouse architecture evolution on Azure Databricks including Delta Lake and Spark.
• Oversee data quality frameworks, monitoring, and SLA adherence.
• Ensure end-to-end ETL/streaming pipelines are scalable and reliable.

Skills

Leadership

Azure Databricks

Python

SQL

Data pipelines

Data quality

Communication

Mentorship

Technical depth

Education

Bachelor's degree

Tools

Databricks

Delta Lake

Apache Spark

Unity Catalog

Databricks Workflows

ADLS Gen2

Azure DevOps

Job description

Randstad Digital Americas in Chicago is seeking a Data Engineering Manager to lead a scrum team of 5-7 data engineers and consultants, delivering the enterprise Databricks data platform.

You will partner with Data Architecture and BI to ensure data is ingested, transformed, and prepared for downstream reporting and dashboards, while setting technical standards and driving data quality across pipelines.