Lead Data Engineer – Databricks

TechDigital Group; Bolingbrook (IL); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading tech company is looking for an experienced Data Engineer to architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake. Candidates should possess 10+ years of experience in data engineering or data architecture and 3+ years of hands-on experience with the Databricks platform. Strong skills in Apache Spark, Delta Lake, Databricks SQL, Python, SQL, and GCP services are essential. This role involves collaborating with multiple teams to enable scalable data products and analytics while defining best practices for data governance and observability.

Qualifications

• 10+ years of experience in data engineering or data architecture in Big Data platforms.
• 3+ years hands-on experience with Databricks platform architecture.

Responsibilities

• Architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake.
• Design robust batch and streaming data pipelines leveraging Apache Spark.
• Lead migration of Jobs from other cloud data platforms to Databricks.
• Implement secure data governance, access control, and lineage using Unity Catalog.
• Architect integrations with Google Cloud Platform.
• Optimize performance and manage compute costs through efficient cluster configuration.
• Collaborate with data engineers, analytics teams, and ML engineers.
• Define best practices for data quality, reliability, and observability.
• Provide technical leadership and mentorship to data engineering teams.

Skills

Apache Spark

Delta Lake

Databricks SQL

Python

SQL

GCP Platform services

Job description

Mandatory Skills

Strong expertise in Apache Spark, Delta Lake, Databricks SQL, Python, SQL and GCP Platform services.

Job Description

• 10+ years of experience in data engineering or data architecture in Big Data platforms.
• 3+ years hands-on experience with Databricks platform architecture.
• Strong expertise in Apache Spark, Delta Lake, Databricks SQL, Python, SQL and GCP Platform services.

Responsibilities

• Architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake.
• Design robust batch and streaming data pipelines leveraging Apache Spark, structured streaming, and modern ELT patterns.
• Lead migration of Jobs from other cloud data platform to Databricks.
• Implement secure data governance, access control, and lineage using Unity Catalog.
• Architect integrations with cloud platforms such as Google Cloud Platform.
• Optimize performance and manage compute costs through efficient cluster configuration and Spark workload tuning.
• Collaborate with data engineers, analytics teams, and ML engineers to enable scalable data products and analytics.
• Define best practices for data quality, reliability, and observability across the data platform.
• Provide technical leadership, architecture guidance, and mentorship to data engineering teams.