Azure Databricks Architect

Prodapt; Richardson (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

Prodapt seeks an experienced Azure Databricks Architect to lead a large-scale Palantir Foundry to Databricks Lakehouse migration. You will own end-to-end architecture, design, migration strategy, governance, and delivery of enterprise data, AI/ML, analytics, and reporting solutions on Databricks.

You will define the target-state architecture, assess existing applications, design scalable pipelines, and lead migration activities while establishing CI/CD and governance standards.

Qualifications

• 12+ years of experience in Data Engineering, Data Architecture, or Data Platforms.
• 5+ years of hands-on Databricks experience designing enterprise-scale solutions.
• Strong expertise in Databricks, Spark, SQL, Python, Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.

Responsibilities

• Define and own the target-state Databricks Lakehouse Architecture, including Delta Lake, Unity Catalog, Medallion architecture, security, governance, and integration patterns.
• Assess Palantir Foundry applications and map capabilities to Databricks solutions.
• Design and implement scalable data ingestion, transformation, batch/streaming, and analytics pipelines using Databricks, Spark, Delta Lake, and Python.
• Architect AI/GenAI solutions including RAG, LLMs, document processing, chatbot, and information extraction capabilities.
• Lead migration planning, execution, validation, cutover, and optimization activities.
• Establish CI/CD, Infrastructure as Code, deployment standards, and best practices using Databricks Asset Bundles, Git, and Terraform.
• Drive performance optimization, cost management, data quality, observability, and governance.
• Collaborate with business and technology stakeholders, translate requirements into scalable solutions, and mentor engineering teams.

Skills

Data Engineering

Data Architecture

Technical leadership

Stakeholder management

Mentoring

Education

Bachelor's or Master's in Computer Science/Engineering

Tools

Databricks

Apache Spark (PySpark)

SQL

Python

Delta Lake

Unity Catalog

Databricks Workflows

Terraform

Git

Databricks SQL

Job description

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, we serve telecom and tech companies that are building networks and shaping the digital experiences of tomorrow. We deliver consulting, managed services, and business reengineering. Our strength comes from being both AI-first and Telco-native. Gartner recognizes Prodapt as a Large, Telecom-Native Regional IT Service Provider.

Job Summary:

We are seeking an experienced Azure Databricks Architect to lead a large-scale Palantir Foundry to Databricks Lakehouse migration. The ideal candidate will own the end-to-end architecture, design, migration strategy, technical governance, and delivery of enterprise data, AI/ML, analytics, and reporting solutions on Databricks.

Key Responsibilities:

• Define and own the target-state Databricks Lakehouse Architecture, including Delta Lake, Unity Catalog, Medallion (Bronze/Silver/Gold) architecture, security, governance, and integration patterns.
• Assess existing Palantir Foundry applications and map capabilities to Databricks solutions.
• Design and implement scalable data ingestion, transformation, batch/streaming, and analytics pipelines using Databricks, Spark, Delta Lake, and Python.
• Architect AI/GenAI solutions including RAG, LLMs, document processing, chatbot, and information extraction capabilities.
• Lead migration planning, execution, validation, cutover, and optimization activities.
• Establish CI/CD, Infrastructure as Code, deployment standards, and best practices using Databricks Asset Bundles, Git, and Terraform.
• Drive performance optimization, cost management, data quality, observability, and governance.
• Collaborate with business and technology stakeholders, translate requirements into scalable solutions, and mentor engineering teams.

Required Skills & Experience:

• 12+ years of experience in Data Engineering, Data Architecture, or Data Platforms.
• 5+ years of hands‑on Databricks experience designing enterprise-scale solutions.
• Strong expertise in Databricks, Apache Spark (PySpark), SQL, Python, Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.
• Experience leading large-scale cloud data platform migrations and modernization programs.
• Hands‑on experience with AI/ML, GenAI, RAG, Vector Search, and LLM‑based applications.
• Strong architecture, stakeholder management, and technical leadership skills.
• Experience creating HLD/LLD, conducting design reviews, and leading distributed engineering teams.
• Bachelor's or Master's degree in Computer Science, Engineering, IT, or related field.

Preferred Experience:

• Prior experience migrating from Palantir Foundry or other enterprise data platforms (Teradata, Oracle, Hadoop, Informatica, SQL Server, etc.) to Databricks.
• Experience working directly with US customers and leading enterprise transformation initiatives.