Remote Senior Data Warehouse Engineer - Spark & PySpark

Binance; Wagga Wagga; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Learning budget

Performance bonus

Wellbeing support

Remote-friendly

Job summary

Halo Labs is hiring a Senior Data Warehouse Engineer to design and run scalable data pipelines on Azure and AWS. You will build PySpark-based processing on Databricks, implement secure data integrations, and partner with software engineers to embed data services into a SaaS product.

You will lead data modeling, observability, and CI/CD practices, ensuring production stability while optimizing cost and performance.

Qualifications

• Hands-on Azure Databricks experience with data engineering patterns.
• Strong Python, PySpark, and SQL for large-scale pipelines.
• Production-grade ETL/ELT pipelines on cloud platforms.
• Experience with SaaS/product engineering environments.
• Knowledge of data lakes and lakehouse architectures.
• Familiarity with DevOps, Git workflows, and CI/CD automation.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks.
• Architect and implement scalable ETL/ELT pipelines on Azure and AWS.
• Build distributed data processing using Python and PySpark on Databricks.
• Develop ingestion pipelines integrating healthcare APIs and third-party systems.
• Design secure data integration patterns (REST APIs, SFTP, webhooks).
• Embed data services into the SaaS product architecture.
• Contribute to backend design for data layer scalability and observability.
• Implement CI/CD pipelines for automated deployment and testing of data workloads.
• Apply infra-as-code across Azure and AWS.
• Optimize Spark jobs, clusters, and storage for performance and cost.
• Maintain robust data models and SaaS-oriented schemas.
• Implement data validation, monitoring, and production alerting.
• Provide technical mentorship and enforce engineering standards.

Skills

Azure Databricks

Python

PySpark

SQL

ETL/ELT pipelines

CI/CD

Git workflows

Data Modeling

Cloud architecture

Databricks cluster mgmt

Observability

Healthcare data

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or related field

Tools

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

AWS S3

AWS Lambda

AWS Glue

RDS

IAM

Job description

Halo Labs is hiring a Senior Data Warehouse Engineer to design and run scalable data pipelines on Azure and AWS. You will build PySpark-based processing on Databricks, implement secure data integrations, and partner with software engineers to embed data services into a SaaS product.

You will lead data modeling, observability, and CI/CD practices, ensuring production stability while optimizing cost and performance.