Senior Data Engineer, Regulatory Systems

State Government of Victoria, Australia; City of Melbourne; On-site; AUD 120,000 - 160,000; Full time

Job summary

The Department of Government Services in Melbourne is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines powering trusted analytics and regulatory intelligence across CAV and the wider group. You will lead data platform initiatives using Microsoft Fabric, Azure, and Databricks, ensuring data reliability and AI readiness across multiple datasets.

You will develop lakehouse/warehouse architectures, curate datasets, enforce data quality and governance, and

Qualifications

• Demonstrated experience designing, developing and delivering scalable data engineering solutions using Microsoft Azure, Microsoft Fabric and/or Databricks, with expertise in Data Factory, Fabric Data Engineering, Spark/PySpark, Delta Lake, Lakehouse, Data Warehouse and advanced SQL.
• Demonstrated expertise in dimensional and Data Vault modelling, Medallion architecture, data contracts, data quality management, metadata and lineage governance, monitoring, observability and performance optimisation practices.
• Experience integrating data platforms with Microsoft Power Platform (Dataverse) and Microsoft Fabric ecosystem.
• Proficiency in Python and related data engineering frameworks for automation, orchestration and reusable solution development.
• Experience implementing CI/CD and automated deployment pipelines, DevOps, DataOps using Azure DevOps and/or GitHub.
• Experience implementing data governance, security and compliance frameworks, including Microsoft Purview, data classification, lineage and access management practices.
• Exposure to AI/ML-enabled data pipelines and platforms supporting ML development and deployment is desirable.
• Relevant Microsoft certifications and experience working within Agile delivery and DevOps frameworks is desirable.

Responsibilities

• Design, build, and maintain scalable data pipelines for batch and near real-time ingestion and transformation across internal and external data sources using Microsoft Fabric and Azure services.
• Implement and maintain Lakehouse and Warehouse solutions using standard data formats and Medallion architecture to support analytics, reporting, and advanced use cases.
• Develop curated datasets and semantic models optimised for consumption, ensuring consistency, usability, and alignment with enterprise standards.
• Implement data quality, validation, monitoring, and testing practices to ensure data accuracy, availability, and reliability, including troubleshooting and root cause analysis.
• Apply data governance and security controls, including access management, data protection, and metadata management aligned with enterprise and regulatory requirements and governance frameworks.
• Contribute to CI/CD processes, version control, and environment management to enable consistent and reliable deployment of data platform assets.
• Collaborate with analysts, data scientists, and business stakeholders to translate requirements into data solutions, reusable components, and data products.
• Contribute to documentation, reusable engineering patterns, and continuous improvement, and support uplift of team capability through knowledge sharing and mentoring.

Skills

Cloud data engineering

Azure

Microsoft Fabric

Databricks

Data Factory

Python

CI/CD / DevOps

Data governance

Dimensional modeling

SQL

Power Platform / Dataverse

Education

Tertiary qualification in IT or data-related field

Tools

Azure

Microsoft Fabric

Databricks

Data Factory

Spark/PySpark

Delta Lake

Lakehouse

Power Platform (Dataverse)

Microsoft Purview

Job description

Location: Melbourne - CBD and Inner Metro suburbs

Reference: 11712

About Us

The Department of Government Services (DGS) wasestablishedon 1 January 2023 to improveeveryone’s experience of doing business and interacting with the Victorian government. We bring important day-to-day services together in one department to make things easy and seamless for Victorians and businesses. We are doing this by connecting and digitising our systems and platforms across state and localgovernment as well as streamlining our corporate and procurement services.

The Senior Data Engineer is a key member of the Regulatory Data Insights team, responsible for designing, building, and maintaining data platforms and pipelines that power trusted analytics and regulatory intelligence across CAV and the wider group. Reporting to the Technical Data Lead, Regulatory Insights and Systems, the role leads complex engineering initiatives across Microsoft Fabric, Azure, and over time Databricks, ensuring regulatory, compliance, and operational datasets are reliable, secure, and ready for analytics and AI.

The position enhances CAV’s data foundations by engineering Lakehouses/Warehouses in OneLake, developing scalable ingestion and transformation pipelines, and implementing reusable data engineering patterns that integrate multiple internal and external systems. This enables high‑quality insights, dashboards, and AI‑ready data products that support policy development, regulatory strategy, compliance intelligence, operational improvement, and public reporting.

The Senior Data Engineer provides expert technical advice on data architecture, performance, and governance; and contributes to standards, CI/CD pipelines, testing frameworks, and documentation that underpin the reliability of the Regulatory Data Insights function.

Working closely with the Enterprise Architect, Senior Data Analysts, ICT, and broader regulatory and intelligence teams, the Senior Data Engineer champions best‑practice data engineering, modern cloud patterns, and scalable platform design that strengthens the department’s analytical and AI readiness. The role also supports the future introduction of Databricks, helping ensure interoperability and consistent engineering standards across platforms.

Key Accountabilities

• Design, build, and maintain scalable data pipelines for batch and near real-time ingestion and transformation across internal and external data sources using Microsoft Fabric and Azure services.
• Implement and maintain Lakehouse and Warehouse solutions using standard data formats and Medallion architecture to support analytics, reporting, and advanced use cases.
• Develop curated datasets and semantic models optimised for consumption, ensuring consistency, usability, and alignment with enterprise standards.
• Implement data quality, validation, monitoring, and testing practices to ensure data accuracy, availability, and reliability, including troubleshooting and root cause analysis.
• Apply data governance and security controls, including access management, data protection, and metadata management aligned with enterprise and regulatory requirements and governance framworks.
• Contribute to CI/CD processes, version control, and environment management to enable consistent and reliable deployment of data platform assets.
• Collaborate with analysts, data scientists, and business stakeholders to translate requirements into data solutions, reusable components, and data products.
• Contribute to documentation, reusable engineering patterns, and continuous improvement, and support uplift of team capability through knowledge sharing and mentoring.

About You

We are seeking applicants who have demonstrated experience from the following criteria:

• Demonstrated experience designing, developing and delivering scalable data engineering solutions using Microsoft Azure, Microsoft Fabric and/or Databricks, having expertise in Data Factory, Fabric Data Engineering, Spark/PySpark, Delta Lake, Lakehouse, Data Warehouse and advanced SQL.
• Demonstrated expertise in dimensional and Data Vault modelling, Medallion architecture, data contracts, data quality management, metadata and lineage governance, monitoring, observability and performance optimisation practices.
• Experienced in integrating data platforms with Microsoft Power Platform (Dataverse), and Microsoft Fabric ecosystem.
• Demonstrated programming experience using Python and related data engineering frameworks for automation, orchestration and reusable solution development.
• Demonstrated experience implementing CI/CD and automated deployment pipelines, DevOps, DataOps using Azure DevOps and/or GitHub.
• Experience implementing data governance, security and compliance frameworks, including Microsoft Purview, data classification, lineage and access management practices, is highly regarded.
• Exposure to AI/ML-enabled data pipelines, and platforms supporting machine learning development and deployment is desirable.
• Relevant Microsoft certifications and experience working within Agile delivery and DevOps frameworks is desirable.

Mandatory requirements

A tertiary qualification in an information technology or data‑related field and/or substantial experience in data engineering, cloud services, or enterprise data platforms are required.

This role is an ongoing position. This position is only open to applicants with relevant rights to work in Australia.

Other relevant information

To be eligible for appointment to this role, applicants will possess corresponding work rights for the advertised employment period. Appointment to an ongoing role is only available to an Australian/New Zealand citizen or an Australian Permanent Resident. Preferred applicants will be required to undertake pre‑employment screening. DGS actively promotes diversity, inclusion and an equal opportunity workplace. We welcome applicants from all diverse backgrounds, including people with disabilities and Aboriginal and Torres Strait Islander peoples. All roles at DGS can be worked flexibly, however it may differ from role to role.