Databricks Lakehouse Architect — AI/GenAI Migration Lead

Prodapt; Richardson (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

Prodapt seeks an experienced Azure Databricks Architect to lead a large-scale Palantir Foundry to Databricks Lakehouse migration. You will own end-to-end architecture, design, migration strategy, governance, and delivery of enterprise data, AI/ML, analytics, and reporting solutions on Databricks.

You will define the target-state architecture, assess existing applications, design scalable pipelines, and lead migration activities while establishing CI/CD and governance standards.

Qualifications

• 12+ years of experience in Data Engineering, Data Architecture, or Data Platforms.
• 5+ years of hands-on Databricks experience designing enterprise-scale solutions.
• Strong expertise in Databricks, Spark, SQL, Python, Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.

Responsibilities

• Define and own the target-state Databricks Lakehouse Architecture, including Delta Lake, Unity Catalog, Medallion architecture, security, governance, and integration patterns.
• Assess Palantir Foundry applications and map capabilities to Databricks solutions.
• Design and implement scalable data ingestion, transformation, batch/streaming, and analytics pipelines using Databricks, Spark, Delta Lake, and Python.
• Architect AI/GenAI solutions including RAG, LLMs, document processing, chatbot, and information extraction capabilities.
• Lead migration planning, execution, validation, cutover, and optimization activities.
• Establish CI/CD, Infrastructure as Code, deployment standards, and best practices using Databricks Asset Bundles, Git, and Terraform.
• Drive performance optimization, cost management, data quality, observability, and governance.
• Collaborate with business and technology stakeholders, translate requirements into scalable solutions, and mentor engineering teams.

Skills

Data Engineering

Data Architecture

Technical leadership

Stakeholder management

Mentoring

Education

Bachelor's or Master's in Computer Science/Engineering

Tools

Databricks

Apache Spark (PySpark)

SQL

Python

Delta Lake

Unity Catalog

Databricks Workflows

Terraform

Git

Databricks SQL

Job description

Prodapt seeks an experienced Azure Databricks Architect to lead a large-scale Palantir Foundry to Databricks Lakehouse migration. You will own end-to-end architecture, design, migration strategy, governance, and delivery of enterprise data, AI/ML, analytics, and reporting solutions on Databricks.

You will define the target-state architecture, assess existing applications, design scalable pipelines, and lead migration activities while establishing CI/CD and governance standards.