Devops Engineer - Databricks

ELLIOTT MOSS CONSULTING PTE. LTD.; Dublin; On-site; EUR 90,000 - 140,000; Full time

Job summary

ELLIOTT MOSS CONSULTING PTE. LTD.

seeks an experienced Databricks Infrastructure & DevOps Engineer to manage, optimize, and secure enterprise Databricks environments hosted on AWS from Ireland’s Dublin region. You will design and deploy workspaces, automate infrastructure with Terraform and Databricks Asset Bundles, and implement CI/CD pipelines using GitLab while maintaining security governance and compliance.

Qualifications

• Bachelor's degree in Computer Science, IT, Engineering, or related field.
• Minimum 3 years in DevOps, Cloud Infra, or Platform Engineering.
• Hands-on Databricks administration experience.

Responsibilities

• Design, deploy, and administer Databricks workspaces and clusters on AWS.
• Build and manage cloud infrastructure using Terraform and DAB.
• Configure AWS services (S3, IAM, VPC, Security Groups).
• Monitor performance, security, and reliability; manage secrets and keys.
• Develop CI/CD pipelines; manage GitLab-based deployments.
• Implement code quality and security scanning (SonarQube, FOD, Nexus IQ).
• Ensure security governance and regulatory compliance; SSAT/VAPT support.

Skills

Cloud infrastructure

DevOps practices

CI/CD design

Scripting (Python/Bash/PowerShell)

Education

Bachelor's degree

Tools

Databricks Administration

AWS Cloud services

Terraform

GitLab CI/CD

SonarQube

Fortify on Demand

Nexus IQ

CloudSCAPE

Python

PowerShell

Bash

Job description

We are looking for an experienced Databricks Infrastructure & DevOps Engineer to manage, optimize, and secure enterprise Databricks environments hosted on AWS.

The ideal candidate will have strong expertise in cloud infrastructure, DevOps automation, CI/CD pipelines, Infrastructure as Code (IaC), and security governance within highly regulated environments.

Key Responsibilities

• Infrastructure & Platform Management Design, deploy, and administer Databricks workspaces and clusters on AWS.
• Build and manage cloud infrastructure using Terraform and Databricks Asset Bundles (DAB).
• Configure and optimize AWS services including S3, IAM, VPC, Security Groups, and networking.
• Monitor platform performance, availability, scalability, and reliability. Manage secrets, encryption keys, and platform security configurations.
• Implement backup, retention, and disaster recovery strategies. DevOps & CI/CD Design, develop, and maintain CI/CD pipelines for Databricks deployments.
• Manage source control and automated deployments using GitLab.
• Implement code quality and security scanning using SonarQube, Fortify on Demand (FOD), Nexus IQ, and CloudSCAPE.
• Automate infrastructure provisioning and application deployment.
• Support release management and deployment automation across environments.
• Security & Compliance Ensure compliance with government security frameworks and data governance standards.
• Support vulnerability assessments, penetration testing (VAPT), and remediation activities.
• Manage identity and access controls through Central Accounts Management (CAM). Implement security monitoring, vulnerability management, audit logging, and compliance reporting.
• Coordinate security testing, System Security Acceptance Testing (SSAT), and regulatory compliance activities.
• Collaboration & Support Collaborate with Data Engineers, Data Analysts, Cloud Architects, and Security teams to support enterprise data platforms.
• Support multiple Databricks environments and data pipelines.
• Provide technical guidance on infrastructure, deployment automation, and platform optimization.
• Maintain technical documentation, operational procedures, and knowledge-sharing materials.
• Support environment provisioning, onboarding, and training activities.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Minimum 3 years of experience in DevOps, Cloud Infrastructure, or Platform Engineering.
• Hands-on experience administering the Databricks platform.
• Strong experience with AWS Cloud services.
• Proficiency in Terraform and Infrastructure as Code (IaC).
• Experience designing and managing GitLab CI/CD pipelines.
• Strong scripting skills using Python, Bash, or PowerShell.
• Experience implementing security best practices in enterprise cloud environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Management Containerization & Orchestration

Technical Skills Databricks Administration AWS Cloud (S3, IAM, VPC, Security Groups) Terraform Databricks Asset Bundles (DAB) GitLab CI/CD SonarQube Fortify on Demand (FOD) Nexus IQ CloudSCAPE Python / Bash / PowerShell Infrastructure as Code (IaC) Security Monitoring & Compliance Vulnerability

Preferred Qualifications

• Experience working in Government Cloud (GCC) or highly regulated environments.
• Knowledge of government security standards, data classification, and compliance frameworks.
• Experience supporting enterprise data platforms and Databricks environments.
• Familiarity with SSAT, VAPT, PDPIA, and security governance processes.
• Preferred Certifications-Databricks Certified Associate (or higher) AWS Certified DevOps Engineer AWS Certified Solutions Architect (Preferred)