Lead Engineer - Data & AI

M&T Resources; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

M&T Resources is seeking a Lead Data Engineer with deep Databricks expertise to drive an enterprise data and AI transformation. You will design and deliver scalable data platforms and pipelines to support GenAI initiatives and agentic AI capabilities at scale.

You will lead technical delivery, shape architecture, enforce data quality and governance, and partner with stakeholders to ensure production-grade Databricks solutions across batch and streaming workloads on cloud platforms.

Qualifications

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark, PySpark, Python and SQL.
• Strong Delta Lake / Delta Tables experience.
• Experience designing complex data pipelines and platforms.
• Cloud experience, ideally Azure or AWS.
• Strong technical leadership and architecture skills.

Responsibilities

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

Skills

Databricks

Spark

PySpark

Python

SQL

Delta Lake

Lakehouse

GenAI

Leadership

Data pipelines

Tools

Azure

AWS

LangChain

LlamaIndex

LangGraph

MLflow

Databricks Workflows

DLT

Job description

We're looking for a Lead Data Engineer with deep, hands-on Databricks expertise to join a major enterprise data and AI transformation.

You'll lead the design and delivery of scalable data platforms and pipelines supporting GenAI and agentic AI initiatives.

Strong Databricks experience is essential. We're looking for someone who has delivered production-grade solutions in Databricks, not just exposure to the platform.

What You'll Do

• Lead data engineering solutions using Databricks and the Lakehouse platform.
• Build scalable pipelines using Spark, PySpark, Python and SQL.
• Design Delta Lake / Lakehouse architectures across batch and streaming.
• Build data foundations for GenAI, RAG and agentic AI.
• Work with structured and unstructured data.
• Contribute to Vector Search, embeddings and RAG capabilities.
• Drive data quality, governance, security and observability.
• Provide technical leadership and influence architecture and engineering standards.

What We're Looking For

Essential:

• Extensive commercial Data Engineering experience.
• Deep hands-on Databricks experience.
• Strong Spark / PySpark, Python and SQL.
• Strong Delta Lake / Delta Tables experience.
• Experience designing complex data pipelines and platforms.
• Cloud experience, ideally Azure or AWS.
• Strong technical leadership and architecture skills.

Desirable:

• GenAI / LLM experience.
• RAG, Vector Search and embeddings.
• Agentic AI experience.
• LangChain, LangGraph, LlamaIndex or MCP.
• Unity Catalog, MLflow, Databricks Workflows or DLT/Lakeflow.
• The Key Requirement

If you're a strong Data Engineer but only have limited Databricks exposure, this role is unlikely to be suitable. We're specifically looking for candidates who are genuinely strong in Databricks and can operate at Lead level.