Lead Data Engineer

elliott moss consulting pte. ltd.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

elliott moss consulting pte. ltd. seeks an experienced Databricks Architect / Lead Data Engineer to design and deliver enterprise-scale data platforms on Databricks and AWS. You will lead architecture initiatives while performing hands-on engineering for scalable Lakehouse solutions and modern data pipelines.

The role includes enabling AI/ML and Generative AI workloads, setting governance with Unity Catalog, and driving CI/CD, testing, and observability across data platforms.

Qualifications

• 5+ years in data engineering or equivalent
• Experience designing enterprise-scale data platforms on cloud
• Hands-on Databricks architecture or lead data engineer role
• Experience delivering AI/ML/Generative AI workloads on Databricks

Responsibilities

• Design and own Databricks Lakehouse architecture on AWS
• Lead multiple Databricks initiatives and mentor data engineers
• Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake
• Develop batch, streaming, and near real-time data solutions
• Enable AI/ML and Generative AI use cases on Databricks
• Establish governance, security, Unity Catalog, RBAC, and operational controls
• Drive CI/CD, automated testing, deployment automation, and observability
• Troubleshoot Spark workloads for performance and cost

Skills

Python

PySpark

SQL

AI/ML workloads

Spark performance

Governance & security

Observability

Cost optimization

Tools

Databricks

Delta Lake

Unity Catalog

AWS

S3

IAM

Job description

Job Description

We are seeking an experienced Databricks Architect / Lead Data Engineer to lead the design, development, and delivery of enterprise-scale data platforms on Databricks and AWS.

The role combines architecture leadership with hands-on engineering responsibilities, including building scalable Lakehouse solutions, modern data pipelines, and AI/ML workloads.

Key Responsibilities

• Design and own Databricks Lakehouse architecture on AWS. Lead multiple Databricks initiatives and mentor data engineers.
• Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold layers).
• Develop batch, streaming, and near real-time data solutions. Enable AI/ML and Generative AI use cases on Databricks.
• Establish governance, security, Unity Catalog, RBAC, and operational controls. Integrate Databricks with AWS services including S3, IAM, networking, and data platforms.
• Drive CI/CD, automated testing, deployment automation, and observability.
• Troubleshoot and optimize Spark workloads for performance, reliability, and cost. Required Experience In Data Engineering experience.
• And hands-on Databricks experience. as a Databricks Architect or equivalent architecture role.
• delivering AI/ML or Generative AI solutions on Databricks. and AWS experience.