Senior Azure Databricks Platform Engineer

Prodapt; Dallas (TX); On-site; USD 160,000 - 210,000; Full time

Job summary

Prodapt seeks a Senior Azure Databricks Platform Engineer to lead deployment, migration, optimization, and governance of enterprise-scale Databricks on Azure. You will focus on performance, reliability, and automation with extensive production support experience.

The role requires 10+ years in data engineering, strong Python/SQL skills, and deep Azure cloud know-how, including networking, authentication, and Unity Catalog governance.

Qualifications

• 10+ years in Data Engineering / Cloud Data Platforms with Azure Databricks administration and migration experience.
• Bachelor's or Master's degree in Computer Engineering or related field.
• Hands-on Azure Databricks including workflows, jobs, cluster management and serverless compute.
• Strong Azure Cloud experience: virtual networking, firewalls, proxy, and authentication/authorization.
• Programming skills in Python and SQL; ETL/ELT and data engineering best practices.
• Experience with Unity Catalog, data governance, access management and data quality frameworks.

Responsibilities

• Prepare and configure Databricks Serverless environment for enterprise teams.
• Design, deploy, and manage Databricks job clusters and workflows.
• Collaborate to enable AI/ML capabilities across the platform.
• Migrate existing Databricks jobs, workflows, and pipelines to modern environments.
• Troubleshoot Azure networking, proxy, firewall, authentication, and connectivity issues.
• Optimize workloads for performance, scalability, reliability, and cost.
• Enhance monitoring and observability with health tracking of jobs and pipelines.
• Maintain DataSphere health checks and automation processes.
• Implement Unity Catalog and governance/security policies.
• Investigate and resolve production incidents and failed jobs.
• Create Databricks Genie Spaces for migrations and modernization initiatives.

Skills

Azure Databricks administration

Python

SQL

Azure cloud

Databricks

Snowflake

Unity Catalog

ETL/ELT

Data governance

Monitoring & observability

Production support

React

TypeScript

Azure Web Apps

AI/LLM

Education

Bachelor's or Master's in Computer Engineering or related field

Tools

Azure Databricks

Snowflake

Unity Catalog

Azure Web Apps

Entra ID

Job description

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A ServiceNow-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A “Great Place To Work® Certified™” company, Prodapt employs over 6,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan. Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

We are seeking a highly skilled Senior Azure Databricks Platform Engineer to lead the deployment, migration, optimization, and governance of enterprise-scale Databricks platforms on Azure. The ideal candidate brings deep expertise in Databricks architecture, Azure cloud services, data platform administration, and production support, with a strong focus on performance, scalability, reliability, and automation.

KEY RESPONSIBILITIES

• Prepare and configure the Databricks Serverless environment for enterprise teams, ensuring all required libraries, dependencies, and platform components are available and maintained.
• Design, deploy, and manage Databricks job clusters and workflows.
• Collaborate with Databricks and enterprise stakeholders to enable and optimize AI/ML capabilities across the platform.
• Migrate existing Databricks jobs, workflows, and data processing pipelines to modernized environments.
• Troubleshoot and resolve Azure networking, proxy, firewall, authentication, and connectivity issues affecting Databricks and related applications.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Enhance operational monitoring and observability by building solutions that track job execution status, start/end times, duration, failures, and overall health.
• Maintain and enhance the DataSphere health check framework and associated automation processes.
• Implement and support Unity Catalog, access controls, governance policies, and security best practices.
• Investigate and resolve production incidents, failed jobs, and data pipeline issues.
• Create and manage Databricks Genie Spaces for business domains as part of migration and modernization initiatives.

REQUIRED QUALIFICATIONS

• 10+ years in Data Engineering / Cloud Data Platforms, with strong Azure Databricks administration and migration expertise
• Bachelor's or Master's degree in Computer Engineering, or in a related field.
• Strong hands-on experience with Azure Databricks, including:
• Workflows, jobs, and cluster management
• Serverless compute
• Performance tuning and optimization
• Proven experience with Microsoft Azure Cloud, including:
• Virtual networking
• Firewalls and proxy configurations
• Authentication and authorization
• Strong programming skills in Python and SQL.
• Experience with ETL/ELT development, migration projects, and modern data engineering practices, including direct Databricks and Snowflake platform experience.
• Expertise in production support, monitoring, alerting, incident management, and job reliability improvements.
• Experience with Unity Catalog, data governance, access management, and data quality frameworks.

PREFERRED QUALIFICATIONS

• Experience with Databricks AI/ML capabilities and generative AI initiatives, including LLM-powered applications and AI-driven workflows.
• Experience building operational dashboards and health-check automation frameworks.
• Knowledge of SSO, OAuth, Microsoft Entra ID, and identity management solutions.
• Experience developing platform monitoring, observability, and operational support portals.
• Familiarity with enterprise-scale cloud data platforms and governance frameworks.

DESIRED SKILLS

Databricks | Azure Cloud | Python | SQL | Snowflake | Unity Catalog | ETL/ELT | Data Governance | Monitoring & Observability | React | TypeScript | Azure Web Apps | AI/LLM | Entra ID | Production Support