Data Engineer (AI)

Resource Agility Pty; City of Melbourne; On-site; AUD 120,000 - 160,000; Full time

Benefits offered by this job

Professional development

AI‑driven culture

Stable national company

Job summary

Resource Agility Pty is seeking a Data Engineer AI to own the build, reliability and ongoing operation of a data platform on Microsoft Fabric in Melbourne. You will develop pipelines, models and quality controls turning Dynamics 365 BC and other data into a unified, high‑integrity data foundation.

You’ll work with an AI Solutions Engineer to accelerate delivery in a small, progressive team. 3–5 years of data engineering experience and hands‑on Fabric experience are essential, with strong SQL and

Qualifications

• 3–5 years of data engineering experience building and supporting production pipelines.
• Hands‑on experience with Microsoft Fabric stack (Lakehouse, Data Factory pipelines, Dataflows Gen2, notebooks).
• Strong SQL and PySpark for transformation, modeling, and performance tuning.
• Practical experience with medallion architecture and dimensional modelling.
• Comfort with AI coding assistants to accelerate development while validating outputs.
• Clear communication to explain data issues to non‑technical stakeholders.

Responsibilities

• Build, schedule, and maintain Fabric pipelines, notebooks, and dataflows in medallion architecture.
• Design conformed dimensions and fact tables for historical data migrations and backfills.
• Monitor pipeline health, manage incremental loads, and optimize capacity usage.
• Implement automated data quality checks and reconciliations to source data.
• Manage DevOps workflows using Azure DevOps, Git, and Fabric deployment pipelines.

Skills

Data engineering

SQL

PySpark

Dimensional modelling

Medallion architecture

Data quality

AI coding assistants

Communication

Tools

Microsoft Fabric

Azure DevOps

Git

Azure Data Factory

ADLS

Azure Synapse

Job description

Our client has a National Footprint that operates a widespread network of businesses across Australia. They currently have a need for The Data Engineer AI to join the Melbourne Team to help build reliability for their data platform on Microsoft Fabric.

About the Role:

The Data Engineer is a new role created to own the build, reliability and ongoing operation of our client's data platform on Microsoft Fabric. Working with the AI Solutions Engineer this role is responsible for building the pipelines, data models and quality controls that turn Microsoft Dynamics 365 Business Central and other operational data into the business's unified, high-integrity data foundation. This is a newly created position for a Data Engineer to completely own the build, reliability, and ongoing operations working within a small progressive team utilising AI coding assistants to accelerate delivery.

Our client has a National Footprint that operates a widespread network of businesses across Australia. They currently have a need for The Data Engineer AI to join the Melbourne Team to help build reliability for their data platform on Microsoft Fabric.

About the Role:

The Data Engineer is a new role created to own the build, reliability and ongoing operation of our client's data platform on Microsoft Fabric. Working with the AI Solutions Engineer this role is responsible for building the pipelines, data models and quality controls that turn Microsoft Dynamics 365 Business Central and other operational data into the business's unified, high-integrity data foundation. This is a newly created position for a Data Engineer to completely own the build, reliability, and ongoing operations working within a small progressive team utilising AI coding assistants to accelerate delivery.

Immediate Priorities Include:

• Completing the Gold layer KPI set.
• Loading historical data for newly acquired businesses.
• Ensuring seamless data flow during the major Business Central ERP Optimisation Program.

Key Responsibilities:

• Fabric Pipeline Engineering: Build, schedule, and maintain Fabric pipelines, notebooks, and dataflows across the medallion architecture (Bronze, Silver, Gold).
• Data Warehouse Delivery: Design conformed dimensions and fact tables, delivering historical data migrations and backfills to support advanced reporting.
• Operations & Ingestion: Monitor pipeline health, manage incremental loads, handle deletions/CDC, and optimize Fabric capacity consumption.
• Data Quality & Governance: Implement automated data quality checks and reconciliations back to source, providing clean datasets to the Power BI reporting team.
• Modern DevOps: Manage work through Azure DevOps backlogs, utilising Git and Fabric deployment pipelines.
• Experience implementing data quality checks, reconciliation and monitoring so issues are found before the business finds them
• Working knowledge of source control and deployment practices, including Git, Azure DevOps and Fabric deployment pipelines

Ideal Skills & Experience Required:

• Experience: 3-5 years of data engineering experience building and supporting production pipelines.
• Microsoft Fabric: Hands‑on experience with Lakehouse, Data Factory pipelines, Dataflows Gen2, and notebooks (essential).
• Technical Stack: Strong SQL and PySpark (or Spark SQL) skills for data transformation, modelling, and performance tuning.
• Architecture: Practical experience with medallion architecture and dimensional modelling.
• AI Savvy: Comfortable using AI coding assistants (e.g., Claude, GitHub Copilot) to accelerate your development with the judgment to validate the output.
• Communication: Clear communication style with the ability to explain data issues to non‑technical stakeholders.
• Working with Technology vendors and Microsoft implementation partners

Highly Regarded:

• Experience with Microsoft Dynamics 365 Business Central data structures
• Exposure to ERP consolidation or migration programs, including historical data loads
• Background in Azure Data Lake Storage (ADLS), Azure Data Factory, or Azure Synapse.
• Microsoft certifications
• Exposure to retail, healthcare, or supply chain environments.

What's on Offer:

• The chance to play a foundational role in shaping a modern data platform.
• A supportive, tech-forward culture that embraces AI and automation.
• A stable, fast-growing national company that makes a real difference in the community.
• Commitment to your ongoing professional development and training.