Microsoft Fabric & PySpark Data Engineer

The HR Ally; Melbourne; On-site; AUD 129,000 - 221,000; Part time

Job summary

The HR Ally is seeking an experienced Data Engineer for a contract role in Melbourne. The position focuses on Microsoft Fabric, PySpark and Data Lake/Lakehouse architectures to design, develop and implement scalable data pipelines.

Responsibilities include building ETL/ELT workflows, transforming and validating data, and ensuring quality at scale. Proficiency with Azure Data Factory, Azure Databricks and Spark is preferred, along with Agile work experience.

Qualifications

• Hands-on experience with Microsoft Fabric and PySpark in enterprise data pipelines.
• Strong understanding of Data Lake and Lakehouse architectures.
• Experience building and optimising ETL/ELT workflows.
• Ability to design, develop and troubleshoot data engineering solutions independently.

Responsibilities

• Design, develop and maintain data pipelines using Microsoft Fabric.
• Build and optimise ETL/ELT workflows and data integration solutions.
• Develop PySpark scripts for large-scale data processing and transformation.
• Work with Data Lake, Lakehouse architectures and related components.
• Perform data transformation, cleansing, validation and optimisation.
• Troubleshoot performance issues and ensure data quality and reliability.
• Leverage AI-assisted coding tools to improve development efficiency.

Skills

Microsoft Fabric

PySpark

Python

Data Lake architecture

Lakehouse architecture

ETL/ELT development

AI-assisted development tools

Independent design & troubleshooting

Tools

Azure Data Factory

Azure Databricks

Apache Spark

GitHub Copilot

Job description

Microsoft Fabric & PySpark Data Engineer

Location: Melbourne
Employment Type: Contract

About the Role

We are seeking an experienced Data Engineer with strong hands‑on expertise in Microsoft Fabric, PySpark and Data Lake solutions. The ideal candidate should have deep technical knowledge and practical experience in designing, developing and implementing scalable data engineering solutions.

Key Responsibilities

• Design, develop and maintain data pipelines using Microsoft Fabric.
• Build and optimise ETL/ELT workflows and data integration solutions.
• Develop PySpark scripts for large-scale data processing and transformation.
• Work with Data Lake, Lakehouse architecture and related data engineering components.
• Perform data transformation, cleansing, validation and optimisation.
• Troubleshoot performance issues and ensure data quality and reliability.
• Leverage AI-assisted coding tools to improve development efficiency and productivity.

Required Skills

• Strong hands‑on experience with Microsoft Fabric.
• Excellent programming skills in PySpark and Python.
• Solid understanding of Data Lake and Lakehouse architectures.
• Experience developing and optimising data pipelines and ETL/ELT processes.
• Strong technical understanding of data engineering concepts and architecture.
• Practical knowledge of AI-assisted development tools such as GitHub Copilot or similar.
• Ability to independently design, develop and troubleshoot technical solutions.

Preferred Skills

• Experience with Azure Data Factory, Azure Databricks and Spark.
• Knowledge of data modelling and performance optimisation.
• Experience working in Agile development environments.