Databricks Infrastructure and DevOps Engineer

ITCAN PTE. LIMITED; Singapore; On-site; SGD 90,000 - 140,000; Full time

Job summary

ITCAN PTE. LIMITED in Singapore seeks an experienced Databricks Infrastructure and DevOps Engineer to manage Databricks clusters, implement robust CI/CD pipelines, and ensure governance and security compliance in a government-focused GCC environment.

The role requires 3+ years in Databricks administration, strong AWS GCC expertise, and proficiency with Terraform and GitLab CI/CD. You will collaborate with data and security teams to maintain secure, scalable data platforms.

Qualifications

• 3+ years of Databricks platform administration experience.
• A solid understanding of AWS GCC environment.

Responsibilities

• Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment.
• Manage IaC with Terraform or Databricks Asset Bundle (DAB).
• Configure AWS services (S3, IAM, VPC, security groups) for Databricks.
• Ensure compliance with Confidential Cloud Eligible and Sensitive High data.
• Develop and maintain CI/CD pipelines using SHIP-HATS tools, GitLab, and code quality gates.
• Coordinate security scanning with Fortify, SonarQube, and Nexus IQ.
• Oversee security compliance scanning with GovTech and CAM integration.
• Monitor system performance, manage secrets and key rotation, and implement data retention policies.
• Collaborate with Data Engineers, Analysts and security teams; support training environments.

Skills

Databricks administration

AWS GCC

Terraform

GitLab CI/CD

Security tools

Government frameworks

Containerization

Scripting (Python/Bash/PowerShell)

Tools

Terraform

Databricks Asset Bundle (DAB)

GitLab

SonarQube

Fortify on Demand

Nexus IQ

SHIP-HATS

GCSOC monitoring

Job description

Job Description

Position Overview

We are seeking an experienced Databricks Infrastructure and DevOps Engineer The successful candidate will be responsible for managing and optimising the Databricks infrastructure, implementing robust CI/CD pipelines using government-approved tools, and ensuring compliance with stringent security and governance requirements.

Key Responsibilities

Infrastructure Management

• Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment
• Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)
• Configure and optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks
• Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management

• Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem
• Configure GitLab for version control and automated deployment workflows
• Implement code quality gates using SonarQube for static code analysis
• Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment
• Oversee dependency scanning and management using Nexus IQ
• Coordinate with CloudSCAPE for security compliance scanning and remediation

Security and Compliance

• Implement and maintain Automated Baseline Log Review (ABLR) systems
• Coordinate with GovTech Cyber Security Operations Centre (GCSOC) for security monitoring
• Manage Central Accounts Management (CAM) integration for user access control
• Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT) findings.
• Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements
• Implement and maintain Vulnerability Management System (VMS)

Platform Operations

• Monitor system performance and implement optimization strategies
• Manage secrets and encryption key rotation for data protection
• Configure email notifications for data processing status and error alerts
• Implement and maintain data retention policies
• Support SaaS whitelisting processes and approvals
• Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support

• Work closely with Data Engineers, Data Analysts and security technical team.
• Support multiple datamarts
• Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines
• Collaborate with various government divisions and stakeholders
• Support training environment setup for user onboarding and skill development

Qualifications

Technical Skills

• 3+ years of experience with Databricks platform administration and optimization
• Strong expertise in AWS services, particularly in GCC environment
• Proficiency in Infrastructure-as-Code tools, preferably Terraform
• Experience with GitLab CI/CD pipeline development and management
• Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)
• Understanding of government security frameworks and compliance requirements
• Experience with containerization and orchestration technologies
• Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance

• Experience with government security clearance processes
• Knowledge of data classification and handling procedures (Sensitive High)
• Understanding of encryption mechanisms and key management
• Familiarity with vulnerability management and remediation processes
• Experience with security monitoring and incident response procedures

Certifications (Preferred)

• Databricks Certified Associate or higher
• AWS Certified DevOps Engineer

Required Experience

• Minimum 3 years of DevOps/Infrastructure engineering experience
• Experience working in government environments, esp in GCC.
• Experience in agile development methodologies

Personal Attributes

• Good communication and collaboration abilities
• Attention to detail and commitment to security best practices
• Strong documentation and knowledge sharing practices
• Proactive approach to system monitoring and maintenance