Senior Big Data Backend Engineer — Spark & Azure Data Lake

Binance; Darwin; À distance; AUD 120 000 - 160 000; Plein temps

Avantages offerts par ce poste

Learning budget

Performance bonuses

Wellbeing support

Remote-first organisation

Résumé du poste

Halo Labs is a future-focused, end-to-end data solutions firm transforming tomorrow today. We build secure data systems powering SaaS platforms, with cloud-first architectures across Azure and AWS.

You will develop production-grade data pipelines, optimise Spark workloads, and contribute to scalable, observable data services within a dynamic team. Join a remote-first organisation that enables travel to client sites in Australia, with learning budgets, bonuses, and wellbeing support.

Qualifications

• Demonstrated experience developing ETL/ELT processes for complex and/or large data movement, transformation and/or visualisations, particularly in a cloud environment.
• Experience preparing data optimised for query performance in cloud compute engines (Spark, Azure SQL, graph databases).
• Experience working with Engineering, Storage and Analytics services in cloud infrastructure.

Responsabilités

• Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform
• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments
• Build distributed data processing solutions using Python and PySpark on Databricks
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity
• Work closely with Software Engineers to embed data services directly into the SaaS product architecture
• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads
• Apply infrastructure-as-code and environment management best practices across Azure and AWS
• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency
• Design and maintain robust data models, including dimensional models and SaaS-oriented data schemas
• Implement data validation, monitoring, and alerting to ensure pipeline reliability and production stability
• Provide technical mentorship and enforce engineering standards across the analytics and data engineering team

Connaissances

ETL/ELT in cloud

Query performance optimization

Cloud infrastructure analytics

Formation

Degree in Computer Science/Engineering/Data Science

Outils

Azure Data Factory

Databricks

Data Lake

Lakehouse

Description du poste

Halo Labs is a future-focused, end-to-end data solutions firm transforming tomorrow today. We build secure data systems powering SaaS platforms, with cloud-first architectures across Azure and AWS.

You will develop production-grade data pipelines, optimise Spark workloads, and contribute to scalable, observable data services within a dynamic team. Join a remote-first organisation that enables travel to client sites in Australia, with learning budgets, bonuses, and wellbeing support.