Azure Databricks Platform Lead for AI-Driven Data Ops

Prodapt; Dallas (TX); On-site; USD 160,000 - 210,000; Full time

Job summary

Prodapt seeks a Senior Azure Databricks Platform Engineer to lead deployment, migration, optimization, and governance of enterprise-scale Databricks on Azure. You will focus on performance, reliability, and automation with extensive production support experience.

The role requires 10+ years in data engineering, strong Python/SQL skills, and deep Azure cloud know-how, including networking, authentication, and Unity Catalog governance.

Qualifications

• 10+ years in Data Engineering / Cloud Data Platforms with Azure Databricks administration and migration experience.
• Bachelor's or Master's degree in Computer Engineering or related field.
• Hands-on Azure Databricks including workflows, jobs, cluster management and serverless compute.
• Strong Azure Cloud experience: virtual networking, firewalls, proxy, and authentication/authorization.
• Programming skills in Python and SQL; ETL/ELT and data engineering best practices.
• Experience with Unity Catalog, data governance, access management and data quality frameworks.

Responsibilities

• Prepare and configure Databricks Serverless environment for enterprise teams.
• Design, deploy, and manage Databricks job clusters and workflows.
• Collaborate to enable AI/ML capabilities across the platform.
• Migrate existing Databricks jobs, workflows, and pipelines to modern environments.
• Troubleshoot Azure networking, proxy, firewall, authentication, and connectivity issues.
• Optimize workloads for performance, scalability, reliability, and cost.
• Enhance monitoring and observability with health tracking of jobs and pipelines.
• Maintain DataSphere health checks and automation processes.
• Implement Unity Catalog and governance/security policies.
• Investigate and resolve production incidents and failed jobs.
• Create Databricks Genie Spaces for migrations and modernization initiatives.

Skills

Azure Databricks administration

Python

SQL

Azure cloud

Databricks

Snowflake

Unity Catalog

ETL/ELT

Data governance

Monitoring & observability

Production support

React

TypeScript

Azure Web Apps

AI/LLM

Education

Bachelor's or Master's in Computer Engineering or related field

Tools

Azure Databricks

Snowflake

Unity Catalog

Azure Web Apps

Entra ID

Job description

Prodapt seeks a Senior Azure Databricks Platform Engineer to lead deployment, migration, optimization, and governance of enterprise-scale Databricks on Azure. You will focus on performance, reliability, and automation with extensive production support experience.

The role requires 10+ years in data engineering, strong Python/SQL skills, and deep Azure cloud know-how, including networking, authentication, and Unity Catalog governance.