Data Engineer - AI/Automation, PySpark & Cloud ETL

Ngo Consulting Partners; Australia; Hybrid; AUD 120,000 - 170,000; Full time

Benefits offered by this job

dedicated learning budget

performance bonuses

wellbeing support

Job summary

Halo Labs is a data solutions firm seeking a data engineer to design, develop, and maintain reusable PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS. You will build distributed data processing with Python and PySpark on Databricks, and develop batch and near real-time ingestion pipelines from healthcare systems and external platforms.

You will design secure data integration patterns, collaborate with software engineers, and contribute to backend data layer design, with

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks
• Architect scalable ETL/ELT pipelines across Azure and AWS environments
• Build distributed data processing solutions using Python and PySpark on Databricks
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity
• Work closely with Software Engineers to embed data services directly into the SaaS product architecture
• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads
• Apply infrastructure-as-code and environment management best practices across Azure and AWS
• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency
• Design and maintain robust data models, including dimensional models and SaaS-oriented data schemas
• Implement data validation, monitoring, and

Skills

PySpark

Python

Databricks

Azure

AWS

CI/CD

Azure DevOps

Git

Data modeling

Tools

Databricks

Azure DevOps

Git

Azure Pipelines

Job description

Halo Labs is a data solutions firm seeking a data engineer to design, develop, and maintain reusable PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS. You will build distributed data processing with Python and PySpark on Databricks, and develop batch and near real-time ingestion pipelines from healthcare systems and external platforms.

You will design secure data integration patterns, collaborate with software engineers, and contribute to backend data layer design, with