Databricks DevOps Lead

Veriipro; Deerfield (IL); Hybrid; USD 120,000 - 150,000; Full time

Job summary

Veriipro is seeking a highly skilled Databricks DevOps Lead to optimize automation, CI/CD processes, and infrastructure management for data platforms on Azure. The ideal candidate will possess extensive knowledge of Databricks, Spark, and DevOps practices, ensuring seamless deployment and operational effectiveness of distributed systems.

This role also involves coordinating with offshore teams and providing mentorship to junior engineers while adhering to best practices and operational excellence.

Qualifications

• 5+ years of DevOps or Site Reliability Engineering (SRE) experience supporting data platforms or distributed systems.
• Strong expertise in CI/CD pipelines for Databricks, Spark jobs, and Azure data components.
• Hands-on experience with provisioning Azure resources via IaC.

Responsibilities

• Design, implement, and maintain CI/CD pipelines for Databricks, Spark jobs, and Azure data services.
• Develop and maintain Infrastructure as Code (Terraform/Bicep/ARM) to provision and manage Azure resources.
• Build automation scripts for deployment and release management of notebooks, JARs, wheels, and configuration artifacts.
• Implement monitoring, logging, and alerting frameworks for Spark jobs and data pipelines using Azure Monitor, Log Analytics, or Grafana.
• Coordinate with offshore teams to ensure alignment on deployments, releases, and support activities.
• Ensure adherence to DevOps best practices, security standards, and operational excellence.
• Mentor and guide junior team members on DevOps practices, CI/CD pipelines, and infrastructure management.

Skills

Databricks and Apache Spark development & deployment

Azure cloud services: Cosmos DB, ADLS Gen2, Event Hubs, Key Vault, AKS

Infrastructure as Code (IAAS): Terraform, Bicep, ARM templates

DevOps Tools: Azure DevOps, GitHub Actions

Scripting & Automation: Python, PowerShell, Bash

Containerization & Orchestration: Docker, Azure Kubernetes Service (AKS)

Monitoring & Observability: Grafana, Azure Monitor, Log Analytics

Testing & Quality: JUnit, Postman, SonarQube

Job description

We are seeking a highly skilled Databricks DevOps Lead to drive automation, CI/CD, and infrastructure management for our data platforms on Azure. The ideal candidate will have deep experience with Databricks, Spark, Azure services, and DevOps practices, enabling seamless deployment, monitoring, and operational excellence of distributed data systems. This role also includes coordinating with offshore teams and mentoring junior engineers.

Mandatory Skills

• Databricks and Apache Spark development & deployment
• Azure cloud services: Cosmos DB, ADLS Gen2, Event Hubs, Key Vault, AKS
• Infrastructure as Code (IAAS): Terraform, Bicep, ARM templates
• DevOps Tools: Azure DevOps (GitHub, CI/CD pipelines, Boards), GitHub Actions
• Scripting & Automation: Python, PowerShell, Bash
• Containerization & Orchestration: Docker, Azure Kubernetes Service (AKS)
• Monitoring & Observability: Grafana, Azure Monitor, Log Analytics
• Testing & Quality: JUnit, Postman, SonarQube

Preferred / Additional Skills

• 5+ years of DevOps or Site Reliability Engineering (SRE) experience supporting data platforms or distributed systems
• Strong expertise in CI/CD pipelines for Databricks, Spark jobs, and Azure data components
• Hands-on experience with provisioning Azure resources via IaC
• Knowledge of release strategies for notebooks, JARs, wheels, and configuration artifacts
• Understanding of security best practices (managed identities, secrets management, RBAC) for Azure data services
• Exposure to reliability engineering concepts such as error budgets, SLOs, and SLI dashboards

Roles & Responsibilities

• Design, implement, and maintain CI/CD pipelines for Databricks, Spark jobs, and Azure data services
• Develop and maintain Infrastructure as Code (Terraform/Bicep/ARM) to provision and manage Azure resources
• Build automation scripts for deployment and release management of notebooks, JARs, wheels, and configuration artifacts
• Implement monitoring, logging, and alerting frameworks for Spark jobs and data pipelines using Azure Monitor, Log Analytics, or Grafana
• Coordinate with offshore teams to ensure alignment on deployments, releases, and support activities
• Ensure adherence to DevOps best practices, security standards, and operational excellence
• Mentor and guide junior team members on DevOps practices, CI/CD pipelines, and infrastructure management