Data Engineer

IT Alliance; Canberra; Hybrid; AUD 150,000 - 230,000; Full time

Job summary

IT Alliance is seeking a Lead Data Engineer in Canberra, Australia, for a long-term contract. The role focuses on hands-on ETL/ELT development, Azure-based data engineering (ADF, Databricks, Data Lake), Spark optimization, and robust data delivery pipelines.

The candidate will source data from Oracle/Ingres/Azure/SQL Server, prepare data using Parquet/Delta, and leverage Azure DevOps and Power BI to build meaningful visualisations.

Qualifications

• Hands-on ETL/ELT development for large-scale data ingestion.
• Extensive Azure data engineering experience with ADF, Databricks, SQL Server or Data Lake.
• Experience with Spark and cloud query performance optimization.
• Build automated data ingestion and transformation pipelines.
• Data prep using Parquet/Delta formats.
• Experience sourcing data from Oracle, Ingres and Azure/SQL Server.
• Experience with Azure DevOps/Visual Studio.
• Power BI analytics and visualisation skills.
• Desirable: ADF+Databricks, Data Lake/Lakehouse architecture, Azure ML, ArcGIS Enterprise.

Responsibilities

• Lead hands-on data pipeline development for large-scale ingestion and transformation.
• Design and implement Azure-based data engineering solutions using ADF, Databricks, and Data Lake.
• Optimize Spark jobs and cloud query performance.
• Build automated pipelines for data ingestion and transformation.
• Prepare data using Parquet/Delta formats.
• Source/integrate data from Oracle, Ingres, Azure and SQL Server.
• Leverage Azure data services for storage and analytics.
• Utilize Azure DevOps/VS for development workflow.
• Create Power BI dashboards and meaningful visualisations.
• Advise on Data Lake/Lakehouse architecture; Azure ML and ArcGIS Enterprise desirable.

Skills

ETL/ELT development

Azure Data Factory

Databricks

SQL Server

Azure Data Lake

Spark

Parquet/Delta

Oracle data sources

Azure DevOps

Power BI

Azure Machine Learning

ArcGIS Enterprise

Job description

One of our Federal Government Clients is seeking to engage a Lead Data Engineer. This is a long-term contractual position.

Please check below all the job details

Contract Duration: 03 Years (12 months initially + 24 months extension)

Work Location: Canberra (03 Days office/02 Days home per week)

Eligibility: You must need to have Baseline Security Clearance or above.

Position Description

• Strong hands-on ETL/ELT development for complex/large-scale data ingestion and transformation.
• Strong Azure experience, particularly Azure Data Factory (ADF), Databricks, SQL Server and/or Azure Data Lake.
• Spark / distributed computing and optimisation of data for cloud query performance.
• Building automated data ingestion and transformation pipelines/frameworks.
• Data preparation using Parquet and/or Delta formats.
• Experience sourcing/integrating data from Oracle, Ingres, Azure and SQL Server.
• Experience across cloud data engineering, storage and analytics services.
• Azure DevOps / Visual Studio development environment.
• Power BI – ability to analyse complex datasets and develop meaningful visualisations.
• Highly desirable: Strong hands-on ADF + Databricks, Data Lake/Lakehouse architecture, Azure Machine Learning and/or ArcGIS Enterprise.