Lead Engineer - Data & AI

M&T Resources; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to design and deliver scalable data platforms supporting GenAI initiatives. You will guide architecture, build end-to-end pipelines and establish standards for data quality, governance, and security.

You will lead with hands-on capability in Spark, PySpark, Python and SQL, and collaborate to advance Delta Lake and Lakehouse architectures across batch and streaming workloads.

Qualifications

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark / PySpark, Python and SQL.

Responsibilities

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

Skills

Databricks

Spark

PySpark

Python

SQL

Delta Lake

Lakehouse

Data architecture

Technical leadership

Cloud (Azure/AWS)

Tools

Unity Catalog

MLflow

Databricks Workflows

DLT / Lakeflow

LangChain

LlamaIndex

LangGraph

Vector Search

MCP

Job description

We're looking for a Lead Data Engineer with deep, hands-on Databricks expertise to join a major enterprise data and AI transformation. You'll lead the design and delivery of scalable data platforms and pipelines supporting GenAI and agentic AI initiatives. Strong Databricks experience is essential. We're looking for someone who has delivered production-grade solutions in Databricks, not just exposure to the platform.

What You'll Do

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

What We're Looking For

Essential:

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark / PySpark, Python and SQL.
• Strong Delta Lake / Delta Tables experience.
• Experience designing complex data pipelines and platforms.
• Cloud experience, ideally Azure or AWS.
• Strong technical leadership and architecture skills.

Desirable:

• GenAI / LLM experience.
• RAG, Vector Search and embeddings.
• Agentic AI experience.
• LangChain, LangGraph, LlamaIndex or MCP.
• Unity Catalog, MLflow, Databricks Workflows or DLT/Lakeflow.
• The Key Requirement

If you're a strong Data Engineer but only have limited Databricks exposure, this role is unlikely to be suitable. We're specifically looking for candidates who are genuinely strong in Databricks and can operate at Lead level.