Senior Databricks Engineer: Azure DevOps & CI/CD Expert

Medix Technology; United States; On-site; USD 88,000 - 95,000; Full time

Job summary

Medix Technology seeks a Senior Databricks DevOps Engineer to design, build, and optimize an enterprise Databricks environment on Microsoft Azure. You will own platform capabilities, CI/CD, infrastructure automation, and production reliability for scalable data engineering.

The ideal candidate is hands-on with Databricks, Azure DevOps, Python, SQL, and PySpark, supporting the platform through its full development and production lifecycle.

Qualifications

• Advanced hands-on experience with Azure Databricks and its platform capabilities.
• Advanced experience with Azure DevOps, including repositories, pipelines, CI/CD, and release automation.
• Strong Python, SQL, PySpark, and Apache Spark development skills.
• Experience with Delta Lake and production-scale ETL/ELT pipelines.
• Experience deploying and managing Databricks workloads across multiple environments.
• Strong understanding of Databricks workflows, compute, cluster configuration, security, and performance optimization.

Responsibilities

• Design, build, and maintain scalable Databricks solutions within Azure
• Develop and optimize data pipelines using Python, SQL, PySpark, Spark, and Delta Lake
• Build CI/CD pipelines through Azure DevOps for Databricks code, jobs, workflows, and infrastructure
• Automate environment provisioning, configuration, testing, and deployment
• Manage Databricks workflows, compute resources, cluster policies, libraries, and access controls
• Support Unity Catalog, data governance, security, and permissions
• Implement monitoring, logging, alerting, retry processes, and failure recovery
• Troubleshoot performance, deployment, integration, and production issues
• Optimize Spark workloads, cluster utilization, job performance, and platform costs
• Establish version-control, testing, release-management, and engineering standards
• Partner with data engineering, architecture, security, analytics, and infrastructure teams
• Participate in technical design reviews and mentor other engineers

Skills

Azure Databricks

Azure DevOps

Python

SQL

PySpark

Apache Spark

Delta Lake

Git-based SDLC

Tools

Terraform

Databricks CLI

Databricks APIs

Databricks Asset Bundles

Docker

Kubernetes

Job description

Medix Technology seeks a Senior Databricks DevOps Engineer to design, build, and optimize an enterprise Databricks environment on Microsoft Azure. You will own platform capabilities, CI/CD, infrastructure automation, and production reliability for scalable data engineering.

The ideal candidate is hands-on with Databricks, Azure DevOps, Python, SQL, and PySpark, supporting the platform through its full development and production lifecycle.