Head Of Cloud Ai Infrastructure & Platform

Telstra Corporation; Sydney; Hybrid; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Remote-first policy

Professional development opportunities

Job summary

Halo Labs is hiring a data engineering lead to design, develop, and optimise scalable PySpark frameworks and ETL pipelines across Azure and AWS. You will build distributed data solutions on Databricks and integrate diverse healthcare-like data sources within a SaaS platform.

Join a remote-first team with a strong focus on governance, security, and observability while collaborating with software engineers to embed data services into the product.

Qualifications

• Strong hands-on experience with Azure Databricks and Azure Data Factory.
• Proficient in Python, SQL, and PySpark for large-scale processing.
• Experience building production data pipelines in SaaS environments.
• Familiarity with healthcare or regulated data environments.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks for standardised data engineering.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating third-party systems.
• Design secure data integration patterns ensuring compliance and data integrity.
• Collaborate with software engineers to embed data services into the SaaS product.
• Contribute to backend system design for data layer scalability, observability, and performance.
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines.
• Apply infrastructure-as-code and environment management across Azure and AWS.
• Optimise Spark jobs, cluster configurations, and storage strategies.
• Design data models including dimensional SaaS schemas.
• Implement data validation, monitoring, and alerting for reliability.

Skills

Python

PySpark

SQL

Databricks

Azure

AWS

ETL/ELT

Data modeling

DevOps basics

Data governance

Education

Bachelor's degree in Computer Science, Engineering, or Data Science

Tools

Databricks

Azure DevOps

Git

REST APIs

Job description

Halo Labs is a future focused, end-to-end data solutions firm – transforming tomorrow, today. Our intelligent and secure technology systems and data driven solutions drive meaningful outcomes and unlock lasting value.

Why work at Halo Labs?

Leading Innovation: We don’t just solve problems; we illuminate a never-ending stream of innovation.

Exceptional Perks: Enjoy outstanding perks like a dedicated learning budget, performance bonuses, and comprehensive wellbeing support.

Remote-First Organisation: Experience the perks of remote work while having the flexibility to travel to client locations throughout Australia.

Inclusive and Engaging: Celebrate diversity in a welcoming space that thrives on new ideas and open conversations, all in a respectful environment.

Inspiring Origins: With a compelling founder story, we are a customer-focused, culture-first organisation.

About the role

• Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform

• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments

• Build distributed data processing solutions using Python and PySpark on Databricks

• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms

• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity

• Work closely with Software Engineers to embed data services directly into the SaaS product architecture

• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance

• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads

• Apply infrastructure-as-code and environment management best practices across Azure and AWS

• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency

• Design and maintain robust data models, including dimensional models and SaaS-oriented data schemas

• Implement data validation, monitoring, and alerting to ensure pipeline reliability and production stability

• Provide technical mentorship and enforce engineering standards across the analytics and data engineering team

About you

• Strong hands-on experience with Azure services including Databricks, Azure Data Factory, Azure SQL, Azure Storage, and Azure DevOps

• Practical experience with AWS services relevant to modern data platforms (S3, Lambda, RDS, Glue, IAM, etc.)

• Advanced proficiency in Python, SQL, and PySpark for large-scale distributed data processing

• Deep experience configuring and managing Databricks clusters for scalable big data workloads

• Experience building production-ready data pipelines in a SaaS or product-led engineering environment

• Strong understanding of cloud-native data architecture, including data lakes, lakehouse architecture, and modular pipeline design

• Experience integrating with third-party systems via APIs and secure data exchange mechanisms

• Exposure to healthcare or regulated data environments, including handling sensitive data securely

• Strong knowledge of data modelling, metadata management, and data governance principles

• Experience implementing automated testing frameworks for data pipelines

• Solid understanding of DevOps practices including Git workflows, branching strategies, and CI/CD automation

• Degree in Computer Science, Engineering, Data Science, or related technical field

Australia - Sydney
Australia - Remote, QL
Full time

Benefits of Working at Halo Labs

+ Market leader in compensation and equity awards

+ Comprehensive physical and mental wellness programs

+ Competitive vacation and holidays for recharge

+ Paid parental and adoption leaves

+ Professional development opportunities for all employees regardless of level or role

+ Employee Networks, geographic neighborhood groups, and volunteer opportunities to build connections

+ Vibrant office culture with world class amenities

+ Great Place to Work Certified across the globe

We are an equal opportunity employer and value diversity at Halo Labs. We do not discriminate on the basis of race, religion, color, national origin, sex, gender identity, sexual orientation, age, disability status, or veteran status.