Senior Big Data Backend Engineer - Remote, PySpark/Databricks

Binance; Rockhampton Regional; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Dedicated learning budget

Performance bonuses

Wellbeing support

Remote-first culture

Job summary

Halo Labs is a forward-thinking data solutions firm delivering secure and scalable data platforms. We design and implement robust ETL/ELT pipelines in Azure and AWS, using PySpark on Databricks and Python to power SaaS products.

We value cloud-native architectures, data governance, and automated testing while offering a remote-first culture with opportunities to travel to client sites across Australia. A strong data engineering background and mentoring mindset are essential.

Qualifications

• Proficiency in Python, SQL, and PySpark for large-scale data processing.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks across the SaaS platform.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems and healthcare APIs.
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks).
• Mentor engineers and enforce engineering standards across the analytics and data engineering team.

Skills

Python

SQL

PySpark

Distributed data processing

Data governance

Education

Degree in Computer Science, Engineering, Data Science, or related technical field

Tools

Azure Databricks

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

AWS (S3, Lambda, RDS, Glue)

Databricks

Git

Azure Pipelines

Job description

Halo Labs is a forward-thinking data solutions firm delivering secure and scalable data platforms. We design and implement robust ETL/ELT pipelines in Azure and AWS, using PySpark on Databricks and Python to power SaaS products.

We value cloud-native architectures, data governance, and automated testing while offering a remote-first culture with opportunities to travel to client sites across Australia. A strong data engineering background and mentoring mindset are essential.