Remote-First Senior Backend Engineer | Big Data & PySpark

Binance; Gladstone Regional; Remote; AUD 150,000 - 190,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Job summary

Halo Labs is a future-focused data solutions firm delivering secure data platforms across Azure and AWS. We are hiring a strong Data Engineer to design scalable PySpark frameworks, build production-grade ETL/ELT pipelines, and mentor the analytics team.

You will work on APIs, data models, and cloud-native architectures, ensuring robust data governance and observability while contributing to a remote-first culture with travel to client sites across Australia.

Qualifications

• Strong hands-on experience with Azure Databricks, Data Factory, SQL, Storage and DevOps.
• Proficient in Python, SQL and PySpark for large-scale distributed data processing.
• Experience building production-grade data pipelines in a SaaS/product context.
• Familiarity with regulated data environments and data governance.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks across the SaaS platform.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS environments.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines from third-party clinical systems and healthcare APIs.
• Design secure data integration patterns (APIs, SFTP, webhooks) ensuring data integrity.
• Collaborate with Software Engineers to embed data services into the SaaS product architecture.
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for deployment and testing.

Skills

Python

SQL

PySpark

Databricks

Azure DevOps

Azure Data Factory

Azure Storage

Cloud architecture

Education

Degree in Computer Science or related field

Tools

Databricks clusters

AWS services

Job description

Halo Labs is a future-focused data solutions firm delivering secure data platforms across Azure and AWS. We are hiring a strong Data Engineer to design scalable PySpark frameworks, build production-grade ETL/ELT pipelines, and mentor the analytics team.

You will work on APIs, data models, and cloud-native architectures, ensuring robust data governance and observability while contributing to a remote-first culture with travel to client sites across Australia.