Data Engineer: PySpark & ETL on Azure/AWS — Remote-First

Dunstone Bros Plumbing And Roofing; Victoria; Remote; AUD 140,000 - 190,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Job summary

Halo Labs, a remote-first data solutions firm, is seeking a data engineer to design and implement scalable PySpark frameworks for a SaaS platform. You will build ETL/ELT pipelines on Azure and AWS, process data with PySpark on Databricks, and integrate third-party clinical systems and healthcare APIs.

You will collaborate with software engineers to embed data services into the product architecture, optimize data layer performance, and implement CI/CD pipelines for automated deployment.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform
• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments
• Build distributed data processing solutions using Python and PySpark on Databricks
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity
• Work closely with Software Engineers to embed data services directly into the SaaS product architecture
• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads
• Apply infrastructure-as-code and environment management best practices across Azure and AWS
• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency

Job description

Halo Labs, a remote-first data solutions firm, is seeking a data engineer to design and implement scalable PySpark frameworks for a SaaS platform. You will build ETL/ELT pipelines on Azure and AWS, process data with PySpark on Databricks, and integrate third-party clinical systems and healthcare APIs.

You will collaborate with software engineers to embed data services into the product architecture, optimize data layer performance, and implement CI/CD pipelines for automated deployment.