Data Engineer: Microsoft Fabric & PySpark Specialist

The HR Ally; Melbourne; On-site; AUD 129,000 - 221,000; Part time

Job summary

The HR Ally is seeking an experienced Data Engineer for a contract role in Melbourne. The position focuses on Microsoft Fabric, PySpark and Data Lake/Lakehouse architectures to design, develop and implement scalable data pipelines.

Responsibilities include building ETL/ELT workflows, transforming and validating data, and ensuring quality at scale. Proficiency with Azure Data Factory, Azure Databricks and Spark is preferred, along with Agile work experience.

Qualifications

• Hands-on experience with Microsoft Fabric and PySpark in enterprise data pipelines.
• Strong understanding of Data Lake and Lakehouse architectures.
• Experience building and optimising ETL/ELT workflows.
• Ability to design, develop and troubleshoot data engineering solutions independently.

Responsibilities

• Design, develop and maintain data pipelines using Microsoft Fabric.
• Build and optimise ETL/ELT workflows and data integration solutions.
• Develop PySpark scripts for large-scale data processing and transformation.
• Work with Data Lake, Lakehouse architectures and related components.
• Perform data transformation, cleansing, validation and optimisation.
• Troubleshoot performance issues and ensure data quality and reliability.
• Leverage AI-assisted coding tools to improve development efficiency.

Skills

Microsoft Fabric

PySpark

Python

Data Lake architecture

Lakehouse architecture

ETL/ELT development

AI-assisted development tools

Independent design & troubleshooting

Tools

Azure Data Factory

Azure Databricks

Apache Spark

GitHub Copilot

Job description

The HR Ally is seeking an experienced Data Engineer for a contract role in Melbourne. The position focuses on Microsoft Fabric, PySpark and Data Lake/Lakehouse architectures to design, develop and implement scalable data pipelines.

Responsibilities include building ETL/ELT workflows, transforming and validating data, and ensuring quality at scale. Proficiency with Azure Data Factory, Azure Databricks and Spark is preferred, along with Agile work experience.