Remote Databricks Data Architect Azure Lakehouse Lead

Lumenalta; United States; Remote; USD 140,000 - 200,000; Full time

Job summary

Lumenalta is seeking an experienced Data Engineer to guide the design and migration of a large-scale data platform to Databricks on Azure. You will lead medallion architecture implementation, define standards, and ensure production readiness during lift-and-shift migration from Dataiku and Hadoop environments.

You will write Python, PySpark and advanced SQL, collaborate with client teams, mentor engineers, and help transition Power BI dashboards to Databricks dashboards while maintaining

Qualifications

• 7+ years of experience as a Data Engineer, including time as a technical lead or data architect on a data platform team.
• Production experience with Databricks on Azure.
• Strong hands-on experience with Python, PySpark, and advanced SQL.
• Proven experience designing and implementing medallion architectures in production.
• Experience migrating production workloads to a modern lakehouse from Dataiku or Hadoop-based platforms such as Cloudera, including keeping outputs stable through cutover.
• Experience with streaming or event-driven ingestion on Azure and Spark Structured Streaming.

Responsibilities

• Guide the overall solution design for the platform, aligning with the client's priorities.
• Define Databricks patterns and standards, including medallion architecture and pipeline conventions.
• Lead migration of production pipelines from Dataiku/Cloudera/Hadoop to Databricks across silver and gold layers.
• Maintain production continuity during transition with parallel runs and safe cutover.
• Evaluate real-time ingestion from Azure services and Databricks integration approaches.
• Support transition of dashboards to Databricks dashboards.

Skills

Technical leadership

Mentoring

Communication

Fluent English

Tools

Databricks on Azure

Python

PySpark

SQL

Spark Structured Streaming

Databricks ZeroBus

Delta Live Tables

Databricks Dashboards

Power BI

Azure Event Hubs

Azure Event Grid

Azure Data Explorer

Unity Catalog

Job description

Lumenalta is seeking an experienced Data Engineer to guide the design and migration of a large-scale data platform to Databricks on Azure. You will lead medallion architecture implementation, define standards, and ensure production readiness during lift-and-shift migration from Dataiku and Hadoop environments.

You will write Python, PySpark and advanced SQL, collaborate with client teams, mentor engineers, and help transition Power BI dashboards to Databricks dashboards while maintaining