Lead Data Engineer - Databricks - $180,000 - Gain \"Productionised\" Ai

Morgan Consulting Idibu; Melbourne; On-site; AUD 162,000 - 198,000; Full time

Job summary

Morgan Consulting is partnering with a fast growing Melbourne-based organisation to hire a Lead Data Engineer specializing in Databricks on Azure. You will lead and hands-on build data pipelines, support enterprise BI, analytics, and AI initiatives in a retail data landscape.

You will work across platform, analytics enablement, BI delivery, production support, DevOps and governance, collaborating with stakeholders to produce robust, scalable lakehouse solutions and trusted production-ready

Qualifications

• Hands-on experience building Databricks lakehouse solutions on Azure.
• Proficient in PySpark, SQL, and Python for data pipelines.
• Experience with BI and AI workloads in enterprise settings.
• Strong stakeholder collaboration and production support skills.
• Knowledge of data governance, security, and DevOps practices.

Responsibilities

• Deliver scalable data solutions for BI, analytics, and AI use cases.
• Develop data pipelines with Databricks, ADF, and ADLS.
• Design and maintain curated data models for analytics and AI.
• Develop Power BI datasets and semantic models.
• Implement CI/CD, security, and governance across data platforms.
• Balance project work with BAU in retail data landscape.

Skills

Databricks

PySpark

Python

SQL

Azure Databricks

Delta Lake

Unity Catalog

Genie AI

Power BI

Tools

Azure Data Factory

Azure Data Lake Storage

Power BI Service

Azure DevOps

Source Control

CI/CD

Job description

Lead Data Engineer - Databricks - Melbourne - $180,000 - Gain "productionised" Agentic AI and Gen AI

Morgan Consulting are partnering with an exciting and fast growing organisation seeking a Lead Data Engineer to join their Data function. This is an opportunity to gain Agentic and Generative AI "Productionised" experience. This role offers strong Career Development within an organisation which operates with very little red-tape and has a supportive and positive culture.

About the Role:

The Lead Data Engineer is a hands on technical specialist responsible for building and supporting modern Databricks Lakehouse solutions on Azure within a complex retail data landscape. Working across both project initiatives and BAU, the role delivers robust, scalable data pipelines and curated analytical datasets that underpin enterprise BI, advanced analytics, and AI. You will collaborate closely with retail stakeholders, BI developers, and broader technology teams to transform high volume retail data into trusted, production ready insights that support critical business outcomes. This is an opportunity to gain Agentic and Generative AI "Productionised" experience.

What You Will Do:

Platform & Analytics Enablement

• Enable enterprise BI, analytics, and AI driven insights by delivering scalable, high quality data solutions that support retail trading, operational, and customer use cases.
• Build and optimise data pipelines and integrations that underpin reporting, advanced analytics, and AI/ML workloads.
• Ensure analytics and reporting solutions are performant, secure, scalable, and fit for enterprise and retail decision making.

Azure & Databricks Data Engineering

• Design, develop, and evolve modern data engineering solutions on Microsoft Azure, with a strong focus on Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS).
• Develop robust data transformations using PySpark, Databricks SQL, SQL, and Python.
• Design, implement, and maintain curated analytical and dimensional data models aligned to retail BI, analytics, and AI requirements.

BI & Semantic Layer Delivery

• Develop, optimise, and support Power BI datasets, semantic models, and reports to deliver trusted, business ready insights.
• Administer Power BI environments, including workspace management, refresh orchestration, security, and access controls.
• Align BI solutions to best practice data modelling, performance optimisation, and usability standards.

Production Support & Operational Excellence

• Provide BAU support across data platforms, pipelines, BI, and analytics solutions, including issue resolution and root cause analysis.
• Monitor data platforms and pipelines, implementing logging, alerting, and performance tuning.
• Proactively identify and resolve data quality, availability, and performance issues in production environments.

Engineering Standards, DevOps & Security

• Implement and maintain CI/CD pipelines for data and analytics solutions using Azure DevOps.
• Apply cloud security, authentication, and access control best practices across Azure data and analytics services.
• Contribute to design reviews, code reviews, and technical discussions to ensure robust, maintainable, and scalable solutions.
• Support the definition and continuous improvement of data engineering, analytics, and BI standards.

Governance & Collaboration

• Contribute to the design, adoption, and communication of data governance practices across the data platform.
• Partner closely with retail business stakeholders to translate requirements into reliable technical solutions.
• Collaborate with engineering, data science, infrastructure, and DevOps teams to deliver secure, production ready solutions.
• Balance delivery across project initiatives and ongoing BAU activities within a complex retail data environment.

What You Will Bring:

• Strong experience with Databricks, including Unity Catalog, Delta Lake, Databricks SQL, and Genie AI, enabling secure data governance, self service analytics, and AI driven insights across enterprise data platforms.
• Strong hands on Data Engineering experience with SQL, Python, PySpark and Databricks SQL.
• Strong experience designing and delivering data solutions on Microsoft Azure - Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS).
• Proven experience designing and building data warehouses and analytical data models, including scalable lakehouse and dimensional modelling patterns.
• Strong understanding of data integration, analytics platforms, and reporting architectures in enterprise environments.
• Hands on experience administering and developing Power BI solutions, including Dataset and semantic model design, Strong DAX capability, Performance optimisation and governance.
• Experience working in DevOps environments, including - CI/CD pipelines, Source control, Release and environment management.
• General understanding of cloud authentication, security, and access control concepts, including role based access and data security principles.
• Demonstrated ability to deliver both project work and BAU/support, operating confidently in a production data and BI environment.
• Proven "enterprise scale" Data Engineering experience to Lead Data Engineer level.

Why Join:

• Opportunity to gain Agentic and Generative AI "Productionised" experience.
• Strong Career Development.
• Amazing culture with very little red tape.

If you are a Lead Data Engineer with strong stakeholder management, who has experience in a fast-paced retail environment, and is outcome focussed we would love to hear from you.