Senior Data Engineer - PySpark & Cloud ETL for SaaS

Mindrift; Sydney; Remote; AUD 120.000 - 180.000; Vollzeit

Benefits dieser Stelle

Learning budget

Performance bonus

Wellbeing support

Remote-friendly

Zusammenfassung

Halo Labs is seeking a Data Engineer with PySpark expertise to design and maintain scalable data pipelines across Azure and AWS in our SaaS platform. You will build in-house PySpark frameworks, implement ETL/ELT pipelines, and collaborate with software engineers to embed data services into the product architecture, while ensuring data quality, security, and observability.

We offer a remote-first work environment with opportunity for travel to client locations across Australia, a learning budget,

Qualifikationen

• Strong hands-on experience with Azure Databricks and Azure Data Factory.
• Proficiency in Python, SQL, and PySpark for large-scale distributed data processing.
• Experience building production data pipelines in a SaaS or product-led environment.
• Knowledge of cloud-native data architectures (data lakes, lakehouse).

Aufgaben

• Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform.
• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating third‑party clinical systems, healthcare APIs, and external enterprise platforms.
• Design secure data integration patterns (REST APIs, SFTP, event‑driven ingestion, webhooks) ensuring compliance and data integrity.
• Work closely with Software Engineers to embed data services directly into the SaaS product architecture.
• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance.
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads.
• Apply infrastructure‑as‑code and environment management best practices across Azure and AWS.
• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency.
• Design and maintain robust data models, including dimensional models and SaaS‑oriented data schemas.
• Implement data validation, monitoring, and alerting to ensure pipeline reliability and production stability.
• Provide technical mentorship and enforce engineering standards across the analytics and data engineering team.

Kenntnisse

Python

PySpark

SQL

Databricks

Azure DevOps

Azure Data Factory

Azure SQL

Azure Storage

AWS

Cloud architecture

Tools

Databricks

S3

Lambda

RDS

Glue

IAM

Jobbeschreibung

Halo Labs is seeking a Data Engineer with PySpark expertise to design and maintain scalable data pipelines across Azure and AWS in our SaaS platform. You will build in-house PySpark frameworks, implement ETL/ELT pipelines, and collaborate with software engineers to embed data services into the product architecture, while ensuring data quality, security, and observability.

We offer a remote-first work environment with opportunity for travel to client locations across Australia, a learning budget,