Databricks Engineer

Tredence Inc.; Chicago (IL); On-site; USD 140,000 - 190,000; Full time

Job summary

Tredence Inc. is seeking an experienced Databricks Lead Engineer / Technical Lead to drive large-scale data platform modernization initiatives.

The role focuses on Databricks Lakehouse architecture, Delta Lake, and Databricks Workflows, with hands-on migration from Azure Synapse and SQL Server to Databricks. You will lead cross-functional teams, mentor engineers, and partner with Data Architects and stakeholders to define migration strategies, implement governance with Unity Catalog, and

Qualifications

• 8+ years data engineering with cloud focus.
• 4+ years hands-on Databricks development.
• Experience migrating on-prem data warehouses to Databricks.
• Strong leadership and client-facing skills.
• Agile/Scrum delivery experience.

Responsibilities

• Lead design and delivery of enterprise data solutions on Databricks Lakehouse.
• Build scalable data pipelines with Delta Lake and Spark.
• Migrate data warehouses and ETL workloads to Databricks.
• Collaborate with stakeholders to define migration strategies.
• Configure Unity Catalog governance and RBAC.
• Ensure data governance, security, and compliance.
• Provide technical leadership to onshore and offshore teams.
• Act as main technical contact for client stakeholders.
• Conduct code reviews and performance tuning.
• Support deployment automation and platform standardization.

Skills

Databricks Workflows

Databricks Lakehouse

Apache Spark

Azure Synapse migrations

SQL Server ETL

Data Warehouses

ETL/ELT Frameworks

Unity Catalog

RBAC

Data Governance

Cloud Data Eng

Technical Leadership

Client Communication

Distributed Teams

DABs

Azure DevOps

CI/CD

Infrastructure as Code

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

Azure DevOps

Unity Catalog

Job description

We are seeking an experienced Databricks Lead Engineer / Technical Lead to drive large-scale data platform modernization initiatives. The ideal candidate will possess deep expertise in Databricks Lakehouse architecture, Delta Lake, Databricks Workflows, and hands‑on experience migrating enterprise data platforms from Azure Synapse and SQL Server to Databricks. This role requires strong technical leadership, client‑facing communication skills, and the ability to guide cross‑functional teams through complex data transformation projects.

Key Responsibilities

• Lead the design, development, and implementation of enterprise‑scale data solutions using Databricks Lakehouse Platform.
• Design and build scalable data pipelines utilizing Databricks Workflows, Delta Lake, and Spark‑based processing frameworks.
• Drive end‑to‑end migration of data warehouses, ETL pipelines, and reporting workloads from Azure Synapse and SQL Server to Databricks.
• Collaborate with Data Architects, Business Stakeholders, and Delivery Teams to define migration strategies and target‑state architectures.
• Configure and manage Unity Catalog, including catalogs, schemas, tables, governance policies, and access controls.
• Ensure adherence to data governance, security, and compliance requirements across the Databricks ecosystem.
• Provide technical leadership, solution guidance, and mentorship to offshore and onshore development teams.
• Act as the primary technical point of contact for client stakeholders, facilitating discussions, solution reviews, and project planning.
• Conduct code reviews, performance tuning, and optimization of Databricks workloads.
• Support deployment automation and platform standardization initiatives.

Required Skills & Qualifications

Technical Skills

• Strong hands‑on experience with:
• Databricks Workflows
• Databricks Lakehouse Architecture
• Apache Spark (PySpark/Scala preferred)
• Azure Synapse Analytics workloads to Databricks
• SQL Server ETL processes to Databricks
• Enterprise data warehouses and data pipelines
• Expertise in designing and implementing scalable ETL/ELT frameworks.
• Hands‑on experience with Unity Catalog, including:
• Catalog and schema management
• Table creation and management
• Role‑based access control (RBAC)
• Data governance implementation
• Strong understanding of cloud‑based data engineering principles and best practices.

Leadership & Stakeholder Management

• Experience serving as an Onshore Technical Lead for data engineering programs.
• Strong client‑facing communication and stakeholder management skills.
• Ability to work closely with:
• Client Business Stakeholders
• Data Architects
• Program Managers
• Delivery Leads
• Experience leading distributed teams across onshore and offshore locations.

Good to Have Skills

• Experience with Databricks Asset Bundles (DABs).
• Familiarity with Azure DevOps and CI/CD implementation for Databricks deployments.
• Understanding of Infrastructure-as-Code and deployment automation practices.
• Knowledge of retail, restaurant, or hospitality operational data systems such as:
• Toast
• HotSchedules
• Experience with modern data governance and data quality frameworks.

Preferred Experience

• 8+ years of overall Data Engineering experience.
• 4+ years of hands‑on Databricks development and solution design experience.
• Prior experience leading cloud data migration and modernization programs.
• Experience working in Agile/Scrum delivery environments.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.