Senior Data Platform Engineer - PySpark/Databricks (Remote)

Binance; Mackay Regional; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Dedicated learning budget

Performance bonuses

Wellbeing support

Job summary

Halo Labs is a forward-thinking data solutions firm hiring a Data Engineer to design and maintain PySpark frameworks and scalable ETL/ELT pipelines on Databricks across Azure and AWS. You will build secure data integrations, mentor engineers, and contribute to scalable SaaS data architectures.

The role emphasizes production-grade data workflows, CI/CD automation, and governance in a remote-first team spread across Australia.

Qualifications

• Strong hands-on experience with Azure services including Databricks, Azure Data Factory, Azure SQL, Azure Storage, and Azure DevOps.
• Practical experience with AWS services relevant to modern data platforms (S3, Lambda, RDS, Glue, IAM, etc.).
• Advanced proficiency in Python, SQL, and PySpark for large-scale distributed data processing.
• Deep experience configuring and managing Databricks clusters for scalable big data workloads.
• Experience building production-ready data pipelines in a SaaS environment.
• Knowledge of data lakes, lakehouse architectures, and modular pipeline design.
• Experience with API integration and secure data exchange mechanisms.
• Exposure to healthcare or regulated data environments and data governance principles.
• Experience implementing automated testing frameworks for data pipelines.
• DevOps practices including Git workflows, branching, and CI/CD automation.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks for standardized data engineering patterns in the SaaS platform.
• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems and healthcare APIs.
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) with data integrity.
• Collaborate with Software Engineers to embed data services into the SaaS product architecture.
• Contribute to backend design discussions for data layer scalability, observability, and performance.
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing.
• Apply infrastructure-as-code and environment management across Azure and AWS.
• Optimize Spark jobs, cluster configurations, and storage strategies for performance and cost.
• Design and maintain robust data models, including dimensional models and SaaS data schemas.
• Implement data validation, monitoring, and alerting to ensure pipeline reliability.
• Provide technical mentorship and enforce engineering standards across analytics and data engineering.

Skills

Azure Databricks

Python

SQL

PySpark

ETL/ELT pipelines

CI/CD

Databricks clusters

Cloud architecture

Education

Degree in Computer Science

Engineering/Data Science

Tools

Azure Data Factory

Azure DevOps

Azure SQL

Azure Storage

Git

Databricks

Job description

Halo Labs is a forward-thinking data solutions firm hiring a Data Engineer to design and maintain PySpark frameworks and scalable ETL/ELT pipelines on Databricks across Azure and AWS. You will build secure data integrations, mentor engineers, and contribute to scalable SaaS data architectures.

The role emphasizes production-grade data workflows, CI/CD automation, and governance in a remote-first team spread across Australia.