Senior Backend Engineer: Big Data with PySpark (Remote)

Binance; Toowoomba Regional; À distance; AUD 140 000 - 210 000; Plein temps

Résumé du poste

Halo Labs is seeking a hands-on Senior Data Engineer to design and scale PySpark-based ETL/ELT pipelines across Azure and AWS for our SaaS platform. You will build Databricks workloads, integrate healthcare data, and implement robust data models with governance.

You will collaborate with software engineers to embed data services into the product, optimise Spark jobs, and drive CI/CD and IaC across cloud environments.

Qualifications

• Extensive hands-on experience with Azure Databricks and data services.
• Experience with AWS data services for modern data platforms.
• Advanced proficiency in Python, SQL and PySpark for large-scale distributed data processing.
• Experience building production-ready data pipelines in a SaaS or product-led environment.
• Knowledge of data lakehouse architectures and modular pipeline design.
• Experience integrating with third-party systems via APIs and secure data exchange.
• Healthcare or regulated data experience is a plus.
• DevOps practices including Git workflows and CI/CD automation.

Responsabilités

• Design, develop, and maintain reusable in-house PySpark frameworks across the SaaS platform.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS environments.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating healthcare systems and APIs.
• Design secure data integration patterns ensuring data integrity and compliance.
• Collaborate with Software Engineers to embed data services into the product architecture.
• Contribute to backend system design for data layer scalability, observability, and performance.
• Implement CI/CD pipelines for automated deployment and testing of data workloads.

Connaissances

Data modelling

Metadata management

Data governance

Formation

Bachelor's degree in Computer Science, Engineering, Data Science, or related technical field

Outils

Python

SQL

PySpark

Databricks

Azure Data Factory

Azure DevOps

Azure SQL

Azure Storage

AWS

Description du poste

Halo Labs is seeking a hands-on Senior Data Engineer to design and scale PySpark-based ETL/ELT pipelines across Azure and AWS for our SaaS platform. You will build Databricks workloads, integrate healthcare data, and implement robust data models with governance.

You will collaborate with software engineers to embed data services into the product, optimise Spark jobs, and drive CI/CD and IaC across cloud environments.