Data Product Development Data Engineer | NV1+

Paxus; City of Melbourne; On-site; AUD 100,000 - 140,000; Full time

Job summary

Paxus is seeking an experienced Data Product Development Data Engineer in Melbourne to join a large government data engineering environment. You will conform, transform and integrate datasets to build reusable data products on a central platform.

The role emphasizes scalable data solutions using Apache Spark, Python, NiFi, Iceberg and Azure DevOps in a secure enterprise setting. Responsibilities include building data engineering solutions, transforming raw data into structured datasets, and

Qualifications

• At least 1 year of Apache Spark experience using Python.
• At least 1 year of Azure DevOps experience.
• At least 1 year of Apache NiFi experience.
• Experience with Apache Iceberg.
• Current NV1 security clearance or higher.
• Ability to work on-site 3-5 days per week.

Responsibilities

• Build and maintain data engineering solutions within a large enterprise data platform.
• Transform and conform raw data into structured, reusable datasets.
• Integrate multiple datasets to develop data products.
• Develop distributed data processing solutions using Apache Spark and Python.
• Build and maintain integration workflows using Apache NiFi.
• Work with Apache Iceberg to support modern data storage and processing.
• Use Azure DevOps for source control and engineering delivery.
• Develop and optimise transformation and integration logic.
• Troubleshoot and enhance existing data engineering solutions.
• Work with platform, engineering and technical stakeholders to turn data requirements into working solutions.

Skills

Python

Apache Spark

Data integration

Data transformation

NV1 clearance

Tools

Azure DevOps

Apache NiFi

Apache Iceberg

Apache Spark

Job description

Melbourne preferred | On-site 3-5 days per week | NV1 minimum

We are seeking an experienced Data Product Development Data Engineer to join a large and complex Federal Government data engineering environment.

This role is focused on taking raw data brought into a central data platform, conforming and integrating datasets, developing transformation logic and building reusable data products.

You'll work extensively with Apache Spark, Python, Apache NiFi, Apache Iceberg and Azure DevOps, developing scalable data solutions within a secure enterprise environment.

What you'll be doing

• Build and maintain data engineering solutions within a large enterprise data platform.
• Transform and conform raw data into structured, reusable datasets.
• Integrate multiple datasets to develop data products.
• Develop distributed data processing solutions using Apache Spark and Python.
• Build and maintain integration workflows using Apache NiFi.
• Work with Apache Iceberg to support modern data storage and processing.
• Use Azure DevOps for source control and engineering delivery.
• Develop and optimise transformation and integration logic.
• Troubleshoot and enhance existing data engineering solutions.
• Work with platform, engineering and technical stakeholders to turn data requirements into working solutions.

What you'll need

Mandatory:

• At least 1 year of Apache Spark experience using Python.
• At least 1 year of Azure DevOps experience.
• At least 1 year of Apache NiFi experience.
• Experience with Apache Iceberg.
• Current NV1 security clearance or higher.
• Ability to work on-site 3-5 days per week.

Highly regarded:

• NV2 clearance.
• Melbourne-based.

Why consider this opportunity?

This is an opportunity to work within a significant enterprise data platform and focus on the engineering work that turns raw data into usable, integrated data products.

The role would suit an experienced Data Engineer who enjoys solving complex data problems, working with modern distributed processing technologies and building practical data solutions.

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.