Data Engineer- Nv1 Security Clearance

Xpt Software Australia Pty; Melbourne; On-site; AUD 120,000 - 180,000; Full time

Job summary

Xpt Software Australia Pty is seeking a Data Engineer to design and optimize ETL pipelines using Spark, NiFi, and Airflow within Cloudera Data Platform. You will configure CDP environments, manage security and access controls, and work with enterprise identity services such as EntraID.

Desirable experience includes Cloudera Data Visualization, MSSQL & PowerBI integration, and prior Defence sector exposure. Strong problem solving and communication skills are essential for cross-team collaboration.

Qualifications

• Must have hands-on ETL development experience in Spark / Cloudera Data Platform.
• Strong experience with NiFi for ETL workflows within CDP environments.
• Proficient in DDLs and Erwin for data modeling and lineage.
• Experience with Ozone & Iceberg, Hive, Hue and Impala in data pipelines.
• Ability to configure and manage CDP environments and access controls, including enterprise identity services (e.g., EntraID).
• DevOps skills using AzureDevOps / VisualStudio Code are essential.
• Expertise in debugging complex ETL pipelines and various ETL methods (truncate & reload, etc.).
• Experience with Cloudera Data Visualization is a plus.
• Experience with MSSQL and PowerBI is a plus.
• Excellent problem solving, communication and task management skills.
• Past work on large agency data warehouses and data integration projects is required.
• Defence sector experience is desirable.

Responsibilities

• Develop and optimize ETL pipelines using Spark, NiFi and Airflow within Cloudera Data Platform.
• Manage CDP environments, security roles and access controls linked to identity services.
• Debug and design robust ETL methods including truncate & reload, rollback, and auditing.
• Collaborate with data visualisation efforts using Cloudera Data Visualisation, MSSQL and PowerBI when needed.
• Communicate clearly with stakeholders and manage tasks across data engineering projects.
• Leverage DevOps practices via AzureDevOps and Visual Studio Code to streamline deployments.

Skills

ETL pipeline development

Spark

Cloudera Data Platform

Airflow

NiFi

DDL

Erwin

Ozone

Iceberg

Hive

Impal a

Ranger security

Atlas security

EntraID

AzureDevOps

VisualStudio Code

ETL debugging

Cloudera Data Visualization

MSSQL

PowerBI

Problem solving

Communication

Task management

Defence experience

Tools

NiFi

AzureDevOps

VisualStudioCode

PowerBI

MSSQL

Cloudera Data Platform

Airflow

Spark

Job description

Position: Data Engineer

Security clearance: Candidates must hold NV1 clearance

Job Details:

• ETL pipeline development in Spark / Cloudera Data Engineering / Airflow (critical requirement)
• ETL pipeline development using NiFi within Cloudera Data Platform (strong requirement)
• ETL development using DDLs and Erwin (Strong requirement)
• ETL development experience using Ozone & Iceberg, Hive, Hue and Impala (critical requirement)
• Configuration and management of Cloudera Data Platform environments and tenancies, including complex ranger and atlas security & access controls, linked to enterprise identity management services, such as EntraID
• DevOps skills using AzureDevOps / VisualStudio Code (critical requirement)
• Complex ETL pipeline debugging and design of various methods of ETL, including truncate & reload etc (critical requirement)
• Developer / Engineering skills in Cloudera Data Visualisation (nice to have)
• Developer / Engineering / Integration skills in MSSQL & PowerBI (nice to have)
• Excellent problem solving, communication and task management skills (strong requirement)
• Previous experience working on large agency data warehouses and data integration projects (strong requirement)
• Pervious experience working in Defence (desirable)