Sr. Databricks Engineer - 258215

Medix Technology; United States; On-site; USD 88,000 - 95,000; Full time

Job summary

Medix Technology seeks a Senior Databricks DevOps Engineer to design, build, and optimize an enterprise Databricks environment on Microsoft Azure. You will own platform capabilities, CI/CD, infrastructure automation, and production reliability for scalable data engineering.

The ideal candidate is hands-on with Databricks, Azure DevOps, Python, SQL, and PySpark, supporting the platform through its full development and production lifecycle.

Qualifications

• Advanced hands-on experience with Azure Databricks and its platform capabilities.
• Advanced experience with Azure DevOps, including repositories, pipelines, CI/CD, and release automation.
• Strong Python, SQL, PySpark, and Apache Spark development skills.
• Experience with Delta Lake and production-scale ETL/ELT pipelines.
• Experience deploying and managing Databricks workloads across multiple environments.
• Strong understanding of Databricks workflows, compute, cluster configuration, security, and performance optimization.

Responsibilities

• Design, build, and maintain scalable Databricks solutions within Azure
• Develop and optimize data pipelines using Python, SQL, PySpark, Spark, and Delta Lake
• Build CI/CD pipelines through Azure DevOps for Databricks code, jobs, workflows, and infrastructure
• Automate environment provisioning, configuration, testing, and deployment
• Manage Databricks workflows, compute resources, cluster policies, libraries, and access controls
• Support Unity Catalog, data governance, security, and permissions
• Implement monitoring, logging, alerting, retry processes, and failure recovery
• Troubleshoot performance, deployment, integration, and production issues
• Optimize Spark workloads, cluster utilization, job performance, and platform costs
• Establish version-control, testing, release-management, and engineering standards
• Partner with data engineering, architecture, security, analytics, and infrastructure teams
• Participate in technical design reviews and mentor other engineers

Skills

Azure Databricks

Azure DevOps

Python

SQL

PySpark

Apache Spark

Delta Lake

Git-based SDLC

Tools

Terraform

Databricks CLI

Databricks APIs

Databricks Asset Bundles

Docker

Kubernetes

Job description

W-2: $64-$69/HR.

We are seeking a Senior Databricks DevOps Engineer to build, automate, and optimize an enterprise Databricks environment on Microsoft Azure. This role will focus on Databricks platform capabilities, CI/CD, infrastructure automation, production reliability, and scalable data engineering.

The ideal candidate is highly hands-on with Databricks, Azure DevOps, Python, SQL, and PySpark and can support the platform throughout its complete development and production lifecycle.

Responsibilities

• Design, build, and maintain scalable Databricks solutions within Azure
• Develop and optimize data pipelines using Python, SQL, PySpark, Spark, and Delta Lake
• Build CI/CD pipelines through Azure DevOps for Databricks code, jobs, workflows, and infrastructure
• Automate environment provisioning, configuration, testing, and deployment
• Manage Databricks workflows, compute resources, cluster policies, libraries, and access controls
• Support Unity Catalog, data governance, security, and permissions
• Implement monitoring, logging, alerting, retry processes, and failure recovery
• Troubleshoot performance, deployment, integration, and production issues
• Optimize Spark workloads, cluster utilization, job performance, and platform costs
• Establish version-control, testing, release-management, and engineering standards
• Partner with data engineering, architecture, security, analytics, and infrastructure teams
• Participate in technical design reviews and mentor other engineers

Required Qualifications

• Advanced hands-on experience with Azure Databricks and its platform capabilities
• Advanced experience with Azure DevOps, including repositories, pipelines, CI/CD, and release automation
• Strong Python, SQL, PySpark, and Apache Spark development skills
• Experience with Delta Lake and production-scale ETL/ELT pipelines
• Experience deploying and managing Databricks workloads across multiple environments
• Strong understanding of Databricks workflows, compute, cluster configuration, security, and performance optimization
• Experience with infrastructure as code, preferably Terraform
• Experience with automated testing, monitoring, logging, alerting, and production support
• Knowledge of cloud security, identity and access management, secrets management, and data governance
• Strong understanding of SDLC, Git-based development, Agile delivery, and release-management practices

Preferred Qualifications

• Databricks Asset Bundles, Databricks CLI, or Databricks APIs
• Unity Catalog administration and governance
• Azure Key Vault, ADLS Gen2, and related Azure data services
• Docker, Kubernetes, or other container technologies
• dbt or Snowflake
• Databricks or Microsoft Azure certifications
• Experience supporting healthcare, health insurance, clinical, or claims data
• Familiarity with HIPAA or other regulated data environments