Data Engineer- NV1 Security Clearance

XPT Software; Brisbane; On-site; AUD 120,000 - 180,000; Full time

Job summary

XPT Software in Brisbane, Australia is seeking a Data Engineer to design and maintain ETL pipelines using Spark, Cloudera Data Platform, NiFi and Airflow. Strong preference for experience with CDP environments and data security controls.

NV1 clearance is required. You will work with Ozone, Iceberg, Hive, Hue and Impala, and collaborate on large agency data warehouses. Exposure to Defence projects is desirable and expected to contribute to complex ETL debugging and integration tasks.

Qualifications

• ETL pipeline development in Spark and Cloudera Data Platform with Airflow is a must.
• Experience with NiFi in Cloudera Data Platform.
• ETL development with DDLs and Erwin is strongly required.
• ETL development using Ozone and Iceberg, Hive, Hue and Impala is critical.
• Config and management of CDP environments and security/access controls linked to enterprise identity management services such as EntraID.
• DevOps skills using AzureDevOps and Visual Studio Code are critical.
• Complex ETL pipeline debugging and design of various methods of ETL (truncate & reload) is critical.
• Defence domain experience is desirable.
• Excellent problem solving, communication and task management skills are required.
• Previous experience with large agency data warehouses and data integration projects is strong requirement.

Skills

Problem solving

Communication

Task management

Defence domain experience

Tools

Spark

Cloudera Data Platform

Airflow

NiFi

Erwin

Ozone

Iceberg

Hive

Hue

Impala

Ranger

Atlas

EntraID

AzureDevOps

Visual Studio Code

Job description

Position: Data Engineer

Security clearance: Candidates must hold NV1 clearance

Job Details:

• ETL pipeline development in Spark / Cloudera Data Engineering / Airflow (critical requirement)
• ETL pipeline development using NiFi within Cloudera Data Platform (strong requirement)
• ETL development using DDLs and Erwin (Strong requirement)
• ETL development experience using Ozone & Iceberg, Hive, Hue and Impala (critical requirement)
• Configuration and management of Cloudera Data Platform environments and tenancies, including complex ranger and atlas security & access controls, linked to enterprise identity management services, such as EntraID
• DevOps skills using AzureDevOps / VisualStudio Code (critical requirement)
• Complex ETL pipeline debugging and design of various methods of ETL, including truncate & reload etc (critical requirement)
• Developer / Engineering skills in Cloudera Data Visualisation (nice to have)
• Developer / Engineering / Integration skills in MSSQL & PowerBI (nice to have)
• Excellent problem solving, communication and task management skills (strong requirement)
• Previous experience working on large agency data warehouses and data integration projects (strong requirement)
• Pervious experience working in Defence (desirable)