Databricks Lakehouse Lead Architect on AWS

ELLIOTT MOSS CONSULTING PTE. LTD.; Penarth, High Street; On-site; GBP 90,000 - 140,000; Full time

Job summary

Elliott Moss Consulting Pte. Ltd. in the United Kingdom is seeking an experienced Databricks Architect / Lead Data Engineer to design, build, and deliver enterprise-scale data platforms on Databricks and AWS.

You will own the lakehouse architecture, develop scalable data pipelines, and enable AI/ML workloads, including Generative AI use cases, while ensuring governance and security across the data ecosystem.

Qualifications

• Experience designing Databricks Lakehouse architectures on AWS.
• Hands-on engineering with data pipelines using Python, PySpark, SQL and Delta Lake.
• Experience enabling AI/ML/Generative AI workloads on Databricks and integrating with AWS services.

Responsibilities

• Design and own Databricks Lakehouse architecture on AWS; mentor data engineers.
• Build and optimize data pipelines using Python, PySpark, SQL, and Delta Lake; implement Medallion Architecture.
• Develop batch, streaming, and near real-time data solutions; enable AI/ML workloads on Databricks.
• Establish governance, security, Unity Catalog, RBAC, and operational controls; integrate with AWS data platforms.
• Drive CI/CD, automated testing, deployment automation, and observability.
• Troubleshoot Spark workloads for performance, reliability, and cost.
• Deliver AI/ML or Generative AI solutions on Databricks and AWS.

Skills

Databricks

AWS

Python

PySpark

SQL

Delta Lake

Medallion Architecture

AI/ML

Generative AI

Governance

Unity Catalog

RBAC

CI/CD

Observability

Spark tuning

Tools

S3

IAM

Job description

Elliott Moss Consulting Pte. Ltd. in the United Kingdom is seeking an experienced Databricks Architect / Lead Data Engineer to design, build, and deliver enterprise-scale data platforms on Databricks and AWS.

You will own the lakehouse architecture, develop scalable data pipelines, and enable AI/ML workloads, including Generative AI use cases, while ensuring governance and security across the data ecosystem.