Lead Data Engineer - Azure & Databricks (Hybrid Canberra)

IT Alliance; Canberra; Ibrido; AUD 150.000 - 230.000; Tempo pieno

Descrizione del lavoro

IT Alliance is seeking a Lead Data Engineer in Canberra, Australia, for a long-term contract. The role focuses on hands-on ETL/ELT development, Azure-based data engineering (ADF, Databricks, Data Lake), Spark optimization, and robust data delivery pipelines.

The candidate will source data from Oracle/Ingres/Azure/SQL Server, prepare data using Parquet/Delta, and leverage Azure DevOps and Power BI to build meaningful visualisations.

Competenze

• Hands-on ETL/ELT development for large-scale data ingestion.
• Extensive Azure data engineering experience with ADF, Databricks, SQL Server or Data Lake.
• Experience with Spark and cloud query performance optimization.
• Build automated data ingestion and transformation pipelines.
• Data prep using Parquet/Delta formats.
• Experience sourcing data from Oracle, Ingres and Azure/SQL Server.
• Experience with Azure DevOps/Visual Studio.
• Power BI analytics and visualisation skills.
• Desirable: ADF+Databricks, Data Lake/Lakehouse architecture, Azure ML, ArcGIS Enterprise.

Mansioni

• Lead hands-on data pipeline development for large-scale ingestion and transformation.
• Design and implement Azure-based data engineering solutions using ADF, Databricks, and Data Lake.
• Optimize Spark jobs and cloud query performance.
• Build automated pipelines for data ingestion and transformation.
• Prepare data using Parquet/Delta formats.
• Source/integrate data from Oracle, Ingres, Azure and SQL Server.
• Leverage Azure data services for storage and analytics.
• Utilize Azure DevOps/VS for development workflow.
• Create Power BI dashboards and meaningful visualisations.
• Advise on Data Lake/Lakehouse architecture; Azure ML and ArcGIS Enterprise desirable.

Conoscenze

ETL/ELT development

Azure Data Factory

Databricks

SQL Server

Azure Data Lake

Spark

Parquet/Delta

Oracle data sources

Azure DevOps

Power BI

Azure Machine Learning

ArcGIS Enterprise

Descrizione del lavoro

IT Alliance is seeking a Lead Data Engineer in Canberra, Australia, for a long-term contract. The role focuses on hands-on ETL/ELT development, Azure-based data engineering (ADF, Databricks, Data Lake), Spark optimization, and robust data delivery pipelines.

The candidate will source data from Oracle/Ingres/Azure/SQL Server, prepare data using Parquet/Delta, and leverage Azure DevOps and Power BI to build meaningful visualisations.