Databricks DevOps Lead - Azure CI/CD & Infra

Veriipro; Deerfield (IL); Hybrid; USD 120,000 - 150,000; Full time

Job summary

Veriipro is seeking a highly skilled Databricks DevOps Lead to optimize automation, CI/CD processes, and infrastructure management for data platforms on Azure. The ideal candidate will possess extensive knowledge of Databricks, Spark, and DevOps practices, ensuring seamless deployment and operational effectiveness of distributed systems.

This role also involves coordinating with offshore teams and providing mentorship to junior engineers while adhering to best practices and operational excellence.

Qualifications

• 5+ years of DevOps or Site Reliability Engineering (SRE) experience supporting data platforms or distributed systems.
• Strong expertise in CI/CD pipelines for Databricks, Spark jobs, and Azure data components.
• Hands-on experience with provisioning Azure resources via IaC.

Responsibilities

• Design, implement, and maintain CI/CD pipelines for Databricks, Spark jobs, and Azure data services.
• Develop and maintain Infrastructure as Code (Terraform/Bicep/ARM) to provision and manage Azure resources.
• Build automation scripts for deployment and release management of notebooks, JARs, wheels, and configuration artifacts.
• Implement monitoring, logging, and alerting frameworks for Spark jobs and data pipelines using Azure Monitor, Log Analytics, or Grafana.
• Coordinate with offshore teams to ensure alignment on deployments, releases, and support activities.
• Ensure adherence to DevOps best practices, security standards, and operational excellence.
• Mentor and guide junior team members on DevOps practices, CI/CD pipelines, and infrastructure management.

Skills

Databricks and Apache Spark development & deployment

Azure cloud services: Cosmos DB, ADLS Gen2, Event Hubs, Key Vault, AKS

Infrastructure as Code (IAAS): Terraform, Bicep, ARM templates

DevOps Tools: Azure DevOps, GitHub Actions

Scripting & Automation: Python, PowerShell, Bash

Containerization & Orchestration: Docker, Azure Kubernetes Service (AKS)

Monitoring & Observability: Grafana, Azure Monitor, Log Analytics

Testing & Quality: JUnit, Postman, SonarQube

Job description

Veriipro is seeking a highly skilled Databricks DevOps Lead to optimize automation, CI/CD processes, and infrastructure management for data platforms on Azure. The ideal candidate will possess extensive knowledge of Databricks, Spark, and DevOps practices, ensuring seamless deployment and operational effectiveness of distributed systems.

This role also involves coordinating with offshore teams and providing mentorship to junior engineers while adhering to best practices and operational excellence.