Data Engineer - NV1 Cleared | Cloudera ETL Expert

XPT Software; Brisbane; On-site; AUD 120,000 - 180,000; Full time

Job summary

XPT Software in Brisbane, Australia is seeking a Data Engineer to design and maintain ETL pipelines using Spark, Cloudera Data Platform, NiFi and Airflow. Strong preference for experience with CDP environments and data security controls.

NV1 clearance is required. You will work with Ozone, Iceberg, Hive, Hue and Impala, and collaborate on large agency data warehouses. Exposure to Defence projects is desirable and expected to contribute to complex ETL debugging and integration tasks.

Qualifications

• ETL pipeline development in Spark and Cloudera Data Platform with Airflow is a must.
• Experience with NiFi in Cloudera Data Platform.
• ETL development with DDLs and Erwin is strongly required.
• ETL development using Ozone and Iceberg, Hive, Hue and Impala is critical.
• Config and management of CDP environments and security/access controls linked to enterprise identity management services such as EntraID.
• DevOps skills using AzureDevOps and Visual Studio Code are critical.
• Complex ETL pipeline debugging and design of various methods of ETL (truncate & reload) is critical.
• Defence domain experience is desirable.
• Excellent problem solving, communication and task management skills are required.
• Previous experience with large agency data warehouses and data integration projects is strong requirement.

Skills

Problem solving

Communication

Task management

Defence domain experience

Tools

Spark

Cloudera Data Platform

Airflow

NiFi

Erwin

Ozone

Iceberg

Hive

Hue

Impala

Ranger

Atlas

EntraID

AzureDevOps

Visual Studio Code

Job description

XPT Software in Brisbane, Australia is seeking a Data Engineer to design and maintain ETL pipelines using Spark, Cloudera Data Platform, NiFi and Airflow. Strong preference for experience with CDP environments and data security controls.

NV1 clearance is required. You will work with Ozone, Iceberg, Hive, Hue and Impala, and collaborate on large agency data warehouses. Exposure to Defence projects is desirable and expected to contribute to complex ETL debugging and integration tasks.