Databricks Lakehouse Engineer | AI Data Platform Lead

Unison Group; Kuala Lumpur; On-site; MYR 180,000 - 300,000; Full time

Job summary

Unison Group in Kuala Lumpur is seeking a hands-on Databricks engineer to design, build, and operate Lakehouse-based data and AI solutions. You will own implementation activities, mentor teams, and drive best practices in data engineering, governance, and DevOps.

The role focuses on Delta Live Tables, Unity Catalog, and scalable data pipelines, with partnerships across customers and cross-functional teams to deliver AI capabilities. Strong cloud experience in AWS/Azure/GCP is preferred.

Qualifications

• Hands-on Databricks with workspace config, notebooks, compute, jobs, and orchestration.
• Delta Lake, DLT/Lakeflow, Unity Catalog, and Databricks SQL Warehouses.
• Proficiency in Python or Scala with strong SQL for data/ML workflows.
• Experience with AWS, Azure, or GCP cloud environments.
• Terraform, Git, CI/CD, and DevOps/MLOps practices.
• Monitoring, observability, security, and governance for large data platforms.
• Strong problem-solving and clear communication with customers and engineers.

Responsibilities

• Design, build, and maintain scalable data pipelines using Delta Live Tables / Lakeflow Spark Declarative Pipelines, Spark SQL, Python, or Scala.
• Optimize batch ETL, streaming, and ML workloads for performance, cost, scalability, and reliability.
• Configure and administer Databricks workspaces, Unity Catalog, cluster policies, SQL Warehouses, access controls, and secure governed environments.
• Automate infrastructure provisioning and deployments using Terraform, Git, and CI/CD pipelines.
• Establish platform observability, monitoring, alerting, and cost-management practices using Splunk, Prometheus, or CloudWatch.
• Collaborate with customers and cross-functional teams to deliver AI/LLM solutions using MLflow, Mosaic AI, and relevant Databricks model-serving capabilities.
• Troubleshoot production issues, improve operational reliability, and document architecture, engineering standards, and runbooks.
• Provide technical guidance, conduct code reviews, and mentor engineers on Databricks, data engineering, governance, and DevOps best practices.

Skills

Databricks

Delta Lake

DLT Lakeflow

Python

Scala

SQL

Spark

Unity Catalog

AWS

Azure

GCP

Terraform

CI/CD

Git

MLOps

MLflow

Monitoring

Tools

Splunk

Prometheus

CloudWatch

Terraform

Git

Job description

Unison Group in Kuala Lumpur is seeking a hands-on Databricks engineer to design, build, and operate Lakehouse-based data and AI solutions. You will own implementation activities, mentor teams, and drive best practices in data engineering, governance, and DevOps.

The role focuses on Delta Live Tables, Unity Catalog, and scalable data pipelines, with partnerships across customers and cross-functional teams to deliver AI capabilities. Strong cloud experience in AWS/Azure/GCP is preferred.