Data Engineer (Databricks)

ELLIOTT MOSS CONSULTING PTE. LTD.; Dublin; On-site; EUR 140,000 - 210,000; Full time

Job summary

Elliott Moss Consulting Pte. Ltd. is seeking an experienced Databricks Architect / Lead Data Engineer to design, develop, and deliver enterprise-scale data platforms on Databricks and AWS.

The role combines architecture leadership with hands-on engineering responsibilities across modern data pipelines, AI/ML workloads, and governance. Responsibilities include leading Databricks initiatives, mentoring engineers, building scalable pipelines, and optimizing Spark workloads while driving CI/CD,

Qualifications

• Extensive experience designing Databricks Lakehouse architectures on AWS.
• Proven ability to lead data engineering teams and deliver scalable pipelines.
• Strong knowledge of governance, Unity Catalog, and security controls.

Responsibilities

• Design and own Databricks Lakehouse architecture on AWS and guide engineers.
• Build and optimize batch and streaming data pipelines using PySpark and SQL.
• Implement Medallion architecture (Bronze/Silver/Gold) across enterprise datasets.
• Enable AI/ML workloads on Databricks and integrate with AWS services.

Skills

Databricks Lakehouse

PySpark

Python

SQL

Delta Lake

Unity Catalog

RBAC

CI/CD

AWS

Tools

Databricks

AWS

S3

Job description

Job Description

We are seeking an experienced Databricks Architect / Lead Data Engineer to lead the design, development, and delivery of enterprise-scale data platforms on Databricks and AWS.

The role combines architecture leadership with hands-on engineering responsibilities, including building scalable Lakehouse solutions, modern data pipelines, and AI/ML workloads.

Key Responsibilities

Design and own Databricks Lakehouse architecture on AWS. Lead multiple Databricks initiatives and mentor data engineers.

Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold layers).

Develop batch, streaming, and near real-time data solutions. Enable AI/ML and Generative AI use cases on Databricks.

Establish governance, security, Unity Catalog, RBAC, and operational controls. Integrate Databricks with AWS services including S3, IAM, networking, and data platforms.

Drive CI/CD, automated testing, deployment automation, and observability.

Troubleshoot and optimize Spark workloads for performance, reliability, and cost. Required Experience In Data Engineering experience.

And hands-on Databricks experience as a Databricks Architect or equivalent architecture role.

Delivering AI/ML or Generative AI solutions on Databricks and AWS experience.