Remote Data Engineer: PySpark, Databricks & SaaS Pipelines

Halo Labs Pty Ltd; City of Melbourne; Hybrid; AUD 90,000 - 120,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Comprehensive wellbeing support

Job summary

A data solutions firm based in City of Melbourne is seeking a skilled data engineer to design and implement scalable data pipelines across Azure and AWS environments. The ideal candidate will have extensive experience with Azure services, strong proficiency in Python and PySpark, and a passion for driving innovation in data engineering. This role offers the flexibility of remote work and a commitment to diversity and open conversations.

Qualifications

• Hands-on experience with Azure services including Databricks, Azure Data Factory, and Azure SQL.
• Practical experience with AWS services relevant to data platforms.
• Advanced proficiency in Python, SQL, and PySpark for data processing.

Responsibilities

• Design and maintain reusable PySpark frameworks for the SaaS platform.
• Architect scalable ETL/ELT pipelines across Azure and AWS environments.
• Implement CI/CD pipelines for automated deployment and testing of data workloads.

Skills

Azure services

Python

SQL

PySpark

AWS services

CI/CD practices

DevOps practices

Data modeling

Education

Degree in Computer Science, Engineering, Data Science, or related technical field

Tools

Azure Data Factory

Databricks

Azure SQL

Azure DevOps

Job description

A data solutions firm based in City of Melbourne is seeking a skilled data engineer to design and implement scalable data pipelines across Azure and AWS environments. The ideal candidate will have extensive experience with Azure services, strong proficiency in Python and PySpark, and a passion for driving innovation in data engineering. This role offers the flexibility of remote work and a commitment to diversity and open conversations.