Data Product Development Data Engineer | NV1+

Paxus - Technology + Digital Talent; City of Melbourne; On-site; AUD 120,000 - 160,000; Full time

Job summary

Paxus - Technology + Digital Talent is seeking an experienced Data Product Development Data Engineer for a large federal government data environment in Melbourne. The role focuses on conforming and integrating datasets, transforming raw data, and building reusable data products on a central data platform.

You will work with Apache Spark, Python, Apache NiFi, Apache Iceberg and Azure DevOps to deliver scalable data solutions within a secure enterprise environment.

Qualifications

• At least 1 year of Apache Spark experience using Python.
• At least 1 year of Azure DevOps experience.
• At least 1 year of Apache NiFi experience.
• Experience with Apache Iceberg.
• Current NV1 security clearance or higher.
• Ability to work on-site 3-5 days per week.

Responsibilities

• Build and maintain data engineering solutions within a large enterprise data platform.
• Transform and conform raw data into structured, reusable datasets.
• Integrate multiple datasets to develop data products.
• Develop distributed data processing solutions using Apache Spark and Python.
• Build and maintain integration workflows using Apache NiFi.
• Work with Apache Iceberg to support modern data storage and processing.
• Use Azure DevOps for source control and engineering delivery.
• Develop and optimise transformation and integration logic.
• Troubleshoot and enhance existing data engineering solutions.
• Work with platform, engineering and technical stakeholders to turn data requirements into working solutions.

Skills

Apache Spark

Python

Azure DevOps

Apache NiFi

Apache Iceberg

Azure Cloud

Tools

Cloudera exposure

Microsoft SQL Server

Job description

Data Product Development Data Engineer | NV1+

Melbourne preferred | On-site 3-5 days per week | NV1 minimum

We are seeking an experienced Data Product Development Data Engineer to join a large and complex Federal Government data engineering environment.

This role is focused on taking raw data brought into a central data platform, conforming and integrating datasets, developing transformation logic and building reusable data products.

You'll work extensively with Apache Spark, Python, Apache NiFi, Apache Iceberg and Azure DevOps, developing scalable data solutions within a secure enterprise environment.

What you'll be doing

• Build and maintain data engineering solutions within a large enterprise data platform.
• Transform and conform raw data into structured, reusable datasets.
• Integrate multiple datasets to develop data products.
• Develop distributed data processing solutions using Apache Spark and Python.
• Build and maintain integration workflows using Apache NiFi.
• Work with Apache Iceberg to support modern data storage and processing.
• Use Azure DevOps for source control and engineering delivery.
• Develop and optimise transformation and integration logic.
• Troubleshoot and enhance existing data engineering solutions.
• Work with platform, engineering and technical stakeholders to turn data requirements into working solutions.

What you'll need

Mandatory:

• At least 1 year of Apache Spark experience using Python.
• At least 1 year of Azure DevOps experience.
• At least 1 year of Apache NiFi experience.
• Experience with Apache Iceberg.
• Current NV1 security clearance or higher.
• Ability to work on-site 3-5 days per week.

Highly regarded:

• NV2 clearance.
• Melbourne-based.
• Cloudera exposure.
• Microsoft SQL Server experience.
• Azure Cloud experience.

Why consider this opportunity?

This is an opportunity to work within a significant enterprise data platform and focus on the engineering work that turns raw data into usable, integrated data products.

The role would suit an experienced Data Engineer who enjoys solving complex data problems, working with modern distributed processing technologies and building practical data solutions.

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability.