Senior Python Data Engineer — Spark & Databricks (Remote)

Mindrift; Coffs Harbour; A distancia; AUD 140.000 - 190.000; Jornada completa

Descripción de la vacante

Halo Labs in Australia is seeking a hands-on Data Engineer to design, develop, and maintain reusable PySpark pipelines and standardised data engineering patterns across the SaaS platform. You will architect and implement scalable ETL/ELT processes on Azure and AWS, build data models, ensure data security and quality, and collaborate with software engineers to embed data services into the product.

This remote-first role offers a learning budget and wellbeing support, plus opportunities to travel

Formación

• Bachelor's degree in a relevant field.
• Strong Python, SQL, and PySpark skills.
• Hands-on experience with Azure Databricks and Data Factory.
• Experience building production-grade data pipelines in SaaS environments.

Responsabilidades

• Design, develop, and maintain PySpark pipelines across the SaaS platform.
• Architect and implement scalable ETL/ELT across Azure and AWS.
• Build batch and near real-time ingestion pipelines.
• Design secure data integration patterns ensuring data integrity and compliance.
• Collaborate with Software Engineers to embed data services into the product architecture.
• Contribute to CI/CD of data workloads and IaC practices.

Conocimientos

Azure Databricks

Azure Data Factory

Python

PySpark

Databricks

SQL

REST APIs

CI/CD

IaC

Cloud architecture

Data modelling

SaaS data pipelines

Educación

Bachelor's degree in CS/Engineering/Data Science

Herramientas

AWS S3

AWS Glue

AWS Lambda

AWS RDS

IAM

Git

Azure DevOps

Azure Pipelines

Descripción del empleo

Halo Labs in Australia is seeking a hands-on Data Engineer to design, develop, and maintain reusable PySpark pipelines and standardised data engineering patterns across the SaaS platform. You will architect and implement scalable ETL/ELT processes on Azure and AWS, build data models, ensure data security and quality, and collaborate with software engineers to embed data services into the product.

This remote-first role offers a learning budget and wellbeing support, plus opportunities to travel