Data Engineer - Remote, PySpark & Cloud ETL

Mindrift; City of Greater Bendigo; Remote; AUD 150,000 - 190,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Job summary

Halo Labs is hiring a data engineer to design, develop, and maintain PySpark frameworks across the SaaS platform. You will architect scalable ETL/ELT pipelines on Azure and AWS and build distributed data processing with PySpark on Databricks.

You will ingest data from clinical systems and external platforms, ensure secure data exchange, and align with engineering teams to embed data services in the product. Remote-first with travel across Australia.

Qualifications

• Hands-on Azure data engineering with Databricks and Azure Data Factory.
• Strong Python, SQL, and PySpark for large-scale processing.
• Experience building production data pipelines in SaaS/product environments.
• Knowledge of data lakes, lakehouse architectures, and modular pipelines.
• Experience integrating third-party systems via APIs and secure data exchange.
• Healthcare data handling experience and data governance understanding.
• Familiarity with CI/CD using DevOps practices and Git workflows.
• Degree in CS/Engineering/Data Science or related field.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks across the SaaS platform.
• Architect and implement scalable, production-grade ETL/ELT pipelines on Azure and AWS.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop ingestion pipelines integrating clinical systems, healthcare APIs, and enterprise platforms.
• Design secure data integration patterns (REST APIs, SFTP, webhooks) ensuring data integrity.
• Collaborate with Software Engineers to embed data services into the SaaS architecture.
• Contribute to data layer design for scalability, observability, and performance.
• Implement CI/CD pipelines for automated deployment and testing of data workloads.

Skills

Azure Databricks

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

Python

SQL

PySpark

APIs integration

Healthcare data

Education

Degree in Computer Science, Engineering, Data Science, or related field

Tools

Databricks

AWS (S3, Lambda, RDS, Glue, IAM)

Job description

Halo Labs is hiring a data engineer to design, develop, and maintain PySpark frameworks across the SaaS platform. You will architect scalable ETL/ELT pipelines on Azure and AWS and build distributed data processing with PySpark on Databricks.

You will ingest data from clinical systems and external platforms, ensure secure data exchange, and align with engineering teams to embed data services in the product. Remote-first with travel across Australia.