Lead Data Engineer

ELLIOTT MOSS CONSULTING PTE. LTD.; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

Elliott Moss Consulting Pte. Ltd. is seeking an experienced Databricks Architect / Lead Data Engineer to design, develop and deliver enterprise-scale data platforms on Databricks and AWS.

The role blends architecture leadership with hands-on engineering, building scalable Lakehouse solutions, modern data pipelines, and AI/ML workloads while enforcing governance, security, and operational controls.

Qualifications

• Experience as Databricks Architect or lead data engineer in enterprise environments.
• Hands-on Databricks experience with building Lakehouse architectures on AWS.
• Proficiency in Python, PySpark, SQL, and Delta Lake.

Responsibilities

• Design and own Databricks Lakehouse architecture on AWS and mentor engineers.
• Build and optimize data pipelines using Python, PySpark, SQL, Delta Lake; implement Medallion Architecture.
• Develop batch, streaming, and near real-time data solutions; enable AI/ML workloads on Databricks.

Skills

Databricks

Python

PySpark

SQL

Delta Lake

Medallion Architecture

CI/CD

Spark tuning

AI/ML on Databricks

RBAC & IAM

Tools

AWS

S3

Unity Catalog

Job description

Job Description

We are seeking an experienced Databricks Architect / Lead Data Engineer to lead the design, development, and delivery of enterprise-scale data platforms on Databricks and AWS.

The role combines architecture leadership with hands‑on engineering responsibilities, including building scalable Lakehouse solutions, modern data pipelines, and AI/ML workloads.

Key Responsibilities

• Design and own Databricks Lakehouse architecture on AWS. Lead multiple Databricks initiatives and mentor data engineers.
• Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold layers).
• Develop batch, streaming, and near real‑time data solutions. Enable AI/ML and Generative AI use cases on Databricks.
• Establish governance, security, Unity Catalog, RBAC, and operational controls. Integrate Databricks with AWS services including S3, IAM, networking, and data platforms.
• Drive CI/CD, automated testing, deployment automation, and observability.
• Troubleshoot and optimize Spark workloads for performance, reliability, and cost.
• Required Experience In Data Engineering experience.
• And hands‑on Databricks experience. as a Databricks Architect or equivalent architecture role.
• delivering AI/ML or Generative AI solutions on Databricks. and AWS experience.