Data Engineer: PySpark, Databricks & Cloud Pipelines

Certified Group

Wollongong City Council

Remote

AUD 120,000 - 180,000

Full time

3 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Job summary

Halo Labs is seeking a senior data engineer to design, develop, and maintain scalable data pipelines in a remote-first environment. You will build production-grade ETL/ELT workflows on Databricks, Azure and AWS, and collaborate with software engineers to embed data services into the SaaS product architecture.

The role requires strong Python, SQL, and PySpark skills, plus hands-on experience with Azure Databricks, Data Factory, and DevOps practices.

Qualifications

Strong hands-on experience with Azure services including Databricks, Azure Data Factory, Azure SQL, Azure Storage, and Azure DevOps.
Practical experience with AWS services relevant to modern data platforms (S3, Lambda, RDS, Glue, IAM, etc.).
Advanced proficiency in Python, SQL, and PySpark for large-scale distributed data processing.

Responsibilities

Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform.
Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments.
Build distributed data processing solutions using Python and PySpark on Databricks.
Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms.
Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity.