Lead Data Engineer: Databricks & GenAI Platform

M&T Resources; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to design and deliver scalable data platforms supporting GenAI initiatives. You will guide architecture, build end-to-end pipelines and establish standards for data quality, governance, and security.

You will lead with hands-on capability in Spark, PySpark, Python and SQL, and collaborate to advance Delta Lake and Lakehouse architectures across batch and streaming workloads.

Qualifications

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark / PySpark, Python and SQL.

Responsibilities

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

Skills

Databricks

Spark

PySpark

Python

SQL

Delta Lake

Lakehouse

Data architecture

Technical leadership

Cloud (Azure/AWS)

Tools

Unity Catalog

MLflow

Databricks Workflows

DLT / Lakeflow

LangChain

LlamaIndex

LangGraph

Vector Search

MCP

Job description

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to design and deliver scalable data platforms supporting GenAI initiatives. You will guide architecture, build end-to-end pipelines and establish standards for data quality, governance, and security.

You will lead with hands-on capability in Spark, PySpark, Python and SQL, and collaborate to advance Delta Lake and Lakehouse architectures across batch and streaming workloads.