Azure Databricks Devops Engineer

iO Associates; England; Remote; GBP 60,000 - 80,000; Full time

Job summary

A technology consultancy in the United Kingdom is seeking a skilled DevOps Engineer to modernize data operations by implementing CI/CD pipelines for Azure Databricks and Azure Data Factory. The ideal candidate has over 5 years of experience in DevOps, strong communication skills, and a Bachelor's degree in a relevant field. This remote role offers occasional travel opportunities and a chance to shape the future of data services.

Qualifications

• Minimum 5 years' experience in DevOps or related roles.
• Proven experience with Azure DevOps and CI/CD pipeline implementation.
• Strong knowledge of Azure Databricks and Azure Data Factory.

Responsibilities

• Design, build, and maintain CI/CD pipelines for Azure Databricks and ADF.
• Collaborate with data engineers and developers to integrate code changes seamlessly.
• Monitor and troubleshoot CI/CD pipelines, ensuring minimal downtime.

Skills

Azure DevOps

CI/CD pipeline implementation

Scripting/automation tools (PowerShell, Python)

Problem-solving skills

Communication abilities

Education

Bachelor's degree in Computer Science, IT, or related field

Tools

Azure Databricks

Azure Data Factory

Docker

Kubernetes

Job description

DevOps Engineer (Azure Data Services)

About the Role

We are looking for a skilled and innovative DevOps Engineer to join our clients growing team. You'll play a pivotal role in modernising our data operations by designing and implementing Continuous Integration and Continuous Deployment (CI/CD) pipelines for Azure Databricks and Azure Data Factory (ADF). This is a fantastic opportunity to shape the future of our Azure environment, driving automation, efficiency, and reliability across our data services. This role is remote, with very occasional abroad travel (paid for) and falls inside IR35. It will be an inital 6 month contract with a likelihood of extension. We have interview slots available.

Key Responsibilities

• Design, build, and maintain CI/CD pipelines for Azure Databricks and ADF
• Collaborate with data engineers and developers to integrate code changes seamlessly
• Automate deployment processes to reduce manual intervention and boost efficiency
• Monitor and troubleshoot CI/CD pipelines, ensuring minimal downtime
• Implement best practices for asset bundling and version control in Azure DevOps
• Optimize Azure DevOps environments for scalability and performance
• Ensure compliance with security standards across all deployment processes
• Document CI/CD processes, procedures, and configurations

What We're Looking For

• Proven experience with Azure DevOps and CI/CD pipeline implementation
• Proficiency in Databricks Asset Bundles (DABs)
• Strong knowledge of Azure Databricks and Azure Data Factory
• Hands-on experience with scripting/automation tools (PowerShell, Python)
• Experience in selective deployment for ADF pipelines
• Familiarity with containerization/orchestration (Docker, Kubernetes) is a plus
• Excellent problem-solving and troubleshooting skills
• Strong communication and collaboration abilities
• Knowledge of CI/CD security best practices
• Detail-oriented mindset with a focus on high-quality automation solutions
• Bachelor's degree in Computer Science, IT, or related field
• Minimum 5 years' experience in DevOps or related roles