Data Architect - Databricks Migration

Lumenalta; United States; À distance; USD 140 000 - 200 000; Plein temps

Résumé du poste

Lumenalta is seeking an experienced Data Engineer to guide the design and migration of a large-scale data platform to Databricks on Azure. You will lead medallion architecture implementation, define standards, and ensure production readiness during lift-and-shift migration from Dataiku and Hadoop environments.

You will write Python, PySpark and advanced SQL, collaborate with client teams, mentor engineers, and help transition Power BI dashboards to Databricks dashboards while maintaining

Qualifications

• 7+ years of experience as a Data Engineer, including time as a technical lead or data architect on a data platform team.
• Production experience with Databricks on Azure.
• Strong hands-on experience with Python, PySpark, and advanced SQL.
• Proven experience designing and implementing medallion architectures in production.
• Experience migrating production workloads to a modern lakehouse from Dataiku or Hadoop-based platforms such as Cloudera, including keeping outputs stable through cutover.
• Experience with streaming or event-driven ingestion on Azure and Spark Structured Streaming.

Responsabilités

• Guide the overall solution design for the platform, aligning with the client's priorities.
• Define Databricks patterns and standards, including medallion architecture and pipeline conventions.
• Lead migration of production pipelines from Dataiku/Cloudera/Hadoop to Databricks across silver and gold layers.
• Maintain production continuity during transition with parallel runs and safe cutover.
• Evaluate real-time ingestion from Azure services and Databricks integration approaches.
• Support transition of dashboards to Databricks dashboards.

Connaissances

Technical leadership

Mentoring

Communication

Fluent English

Outils

Databricks on Azure

Python

PySpark

SQL

Spark Structured Streaming

Databricks ZeroBus

Delta Live Tables

Databricks Dashboards

Power BI

Azure Event Hubs

Azure Event Grid

Azure Data Explorer

Unity Catalog

Description du poste

About the Role

We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. This engagement is a modernization program that moves production data workloads off legacy Dataiku and Cloudera/Hadoop environments and onto Databricks on Azure.

What You'll Do

• Guide the overall solution design for the platform, working with the client's project lead to align on priorities as they evolve.
• Define the Databricks patterns and standards the joint team follows, including medallion architecture (bronze, silver, gold) and pipeline conventions.
• Lead the migration of a production pipeline from Dataiku and Cloudera/Hadoop to Databricks across the silver and gold layers, following a lift-and-shift approach first and optimization later.
• Keep production outputs running throughout the transition through parallel runs, data reconciliation and safe cutover, since business-critical operations depend on this pipeline.
• Guide the evaluation and design of real-time event ingestion from Azure Event Hubs, Event Grid and Azure Data Explorer into Databricks, comparing approaches such as Spark Structured Streaming and Databricks ZeroBus.
• Support the transition of Power BI dashboards to Databricks dashboards.
• Work hands-on across workstreams, review designs and code, and step in to unblock engineers whenever an issue comes up.
• Pair with the client's engineers and share Databricks practices so their team can own and evolve the platform over time.
• Write clean, efficient, and maintainable Python, PySpark, and advanced SQL.

What We’re Looking For

• 7+ years of experience as a Data Engineer, including time as a technical lead or data architect on a data platform team.
• Production experience with Databricks on Azure.
• Strong, hands-on experience with Python, PySpark, and advanced SQL.
• Proven experience designing and implementing medallion architectures in production.
• Experience migrating production workloads to a modern lakehouse from Dataiku or from Hadoop-based platforms such as Cloudera, including keeping outputs stable through cutover.
• Experience with streaming or event-driven ingestion on Azure, such as Event Hubs, Event Grid, or Azure Data Explorer, and with Spark Structured Streaming.
• Comfortable leading through influence inside a client's team: setting technical direction while the client owns priorities and timeline.
• A track record of mentoring and enabling other engineers, not just directing them.
• Strong communication skills, with the ability to explain design decisions to engineers and stakeholders.
• Fluent English, both written and spoken.

Nice to Have

• Experience migrating workflows specifically from Dataiku to Databricks.
• Experience with Databricks ZeroBus or other direct-to-lakehouse streaming ingestion.
• Experience with Databricks dashboards, or migrating reporting off Power BI.
• Working knowledge of Unity Catalog.
• Experience with Delta Live Tables or Databricks declarative pipelines, Databricks Workflows, and Databricks Asset Bundles.
• Experience with high-volume, real-time event or sensor data.
• Databricks Data Engineer Professional certification.

Why Lumenalta is an amazing place to work at

At Lumenalta, you can expect that you will:

• Be 100% dedicated to one project at a time so that you can innovate and grow.
• Be a part of a team of talented and friendly senior-level developers.
• Work on projects that allow you to use leading tech.

For Databricks practitioners specifically, we are a Databricks partner with Champions and MVPs on our team, and that carries three concrete things:

• Delivery Partner Program qualification. Databricks co-delivers its Professional Services engagements through a short list of approved partner organizations, and practitioners have to be individually qualified before they can work on them. The qualification is granted once at the individual level and does not expire, so it stays with you. Partner organizations are the route in, so the qualified pool stays small. That scarcity is most of the value, and we sponsor it.
• Champion path. Databricks Champion is an individual recognition with its own badge, a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit. Nomination has to come from a partner organization, which puts it out of reach for independents and most contractors. We nominate actively and regularly, and the Champions already on our team mentor candidates through it.
• Certifications covered. We pay the exam fee for any Databricks certification you want to take.

Our Process

A screening call, a technical interview, and a HackerRank assessment.

The technical interview is a conversation rather than a whiteboard exam. We will ask you to walk through a Databricks platform you designed and led and how you set patterns for the team. We will also ask how you migrated a production workload off a legacy platform without breaking downstream consumers, and what you would change. Relevant project experience matters more to us than certifications.

Location

This is a fully remote position.

Must have availability to work U.S. Eastern time zone business hours.

Application Deadline

Applications will be accepted until October 25th, 2026. Candidates can expect feedback by November 2nd, 2026.