Remote Senior Backend Engineer - PySpark & Data

Binance; Alice Springs; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Remote-first with travel to client AU

Job summary

Halo Labs is seeking a data engineer to design, develop and maintain reusable PySpark frameworks across a SaaS platform. You will architect scalable ETL/ELT pipelines on Azure and AWS, and build distributed data processing with Python and PySpark on Databricks.

You will implement secure data integration patterns, collaborate with software engineers to deploy data services, and contribute to backend data layer design with emphasis on performance and observability.

Qualifications

• Hands-on experience with Azure Databricks, ADF, SQL and Storage.
• Proficient in Python, SQL and PySpark for large-scale processing.
• Experience building production data pipelines in SaaS environments.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks.
• Architect scalable ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing using Python and PySpark on Databricks.
• Develop ingestion pipelines integrating healthcare APIs and third-party systems.
• Design secure data integration patterns ensuring data integrity.
• Collaborate with software engineers to embed data services in SaaS product.
• Contribute to backend design for data layer scalability and observability.
• Implement CI/CD pipelines for data workloads using Azure DevOps and Git.
• Apply infrastructure-as-code and environment management across clouds.
• Optimize Spark jobs and storage strategies for performance and cost.
• Design robust data models including dimensional models for SaaS.
• Implement data validation, monitoring, and alerting for pipeline reliability.
• Provide technical mentorship and enforce engineering standards.

Skills

Azure Databricks

Python

SQL

Distributed data processing

Data modeling

DevOps practices

ETL/ELT design

SaaS product experience

Healthcare/regulatory data awareness

Mentorship

Education

Degree in Computer Science, Engineering, Data Science, or related technical field

Tools

Databricks

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

AWS (S3, Lambda, RDS, Glue)

Git

Azure Pipelines

Spark

Job description

Halo Labs is seeking a data engineer to design, develop and maintain reusable PySpark frameworks across a SaaS platform. You will architect scalable ETL/ELT pipelines on Azure and AWS, and build distributed data processing with Python and PySpark on Databricks.

You will implement secure data integration patterns, collaborate with software engineers to deploy data services, and contribute to backend data layer design with emphasis on performance and observability.