Lead Data Engineer: Databricks & GenAI Platform Architect

M&T Resources; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to drive an enterprise data and AI transformation. You will design and deliver scalable data platforms and pipelines to support GenAI initiatives and agentic AI capabilities at scale.

You will lead technical delivery, shape architecture, enforce data quality and governance, and partner with stakeholders to ensure production-grade Databricks solutions across batch and streaming workloads on cloud platforms.

Qualifications

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark, PySpark, Python and SQL.
• Strong Delta Lake / Delta Tables experience.
• Experience designing complex data pipelines and platforms.
• Cloud experience, ideally Azure or AWS.
• Strong technical leadership and architecture skills.

Responsibilities

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

Skills

Databricks

Spark

PySpark

Python

SQL

Delta Lake

Lakehouse

GenAI

Leadership

Data pipelines

Tools

Azure

AWS

LangChain

LlamaIndex

LangGraph

MLflow

Databricks Workflows

DLT

Job description

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to drive an enterprise data and AI transformation. You will design and deliver scalable data platforms and pipelines to support GenAI initiatives and agentic AI capabilities at scale.

You will lead technical delivery, shape architecture, enforce data quality and governance, and partner with stakeholders to ensure production-grade Databricks solutions across batch and streaming workloads on cloud platforms.