Databricks Lakehouse Architect & Data Platform Lead

TechDigital Group; Bolingbrook (IL); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading tech company is looking for an experienced Data Engineer to architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake. Candidates should possess 10+ years of experience in data engineering or data architecture and 3+ years of hands-on experience with the Databricks platform. Strong skills in Apache Spark, Delta Lake, Databricks SQL, Python, SQL, and GCP services are essential. This role involves collaborating with multiple teams to enable scalable data products and analytics while defining best practices for data governance and observability.

Qualifications

• 10+ years of experience in data engineering or data architecture in Big Data platforms.
• 3+ years hands-on experience with Databricks platform architecture.

Responsibilities

• Architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake.
• Design robust batch and streaming data pipelines leveraging Apache Spark.
• Lead migration of Jobs from other cloud data platforms to Databricks.
• Implement secure data governance, access control, and lineage using Unity Catalog.
• Architect integrations with Google Cloud Platform.
• Optimize performance and manage compute costs through efficient cluster configuration.
• Collaborate with data engineers, analytics teams, and ML engineers.
• Define best practices for data quality, reliability, and observability.
• Provide technical leadership and mentorship to data engineering teams.

Skills

Apache Spark

Delta Lake

Databricks SQL

Python

SQL

GCP Platform services

Job description

A leading tech company is looking for an experienced Data Engineer to architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake. Candidates should possess 10+ years of experience in data engineering or data architecture and 3+ years of hands-on experience with the Databricks platform. Strong skills in Apache Spark, Delta Lake, Databricks SQL, Python, SQL, and GCP services are essential. This role involves collaborating with multiple teams to enable scalable data products and analytics while defining best practices for data governance and observability.