Senior Data Warehouse Engineer — Big Data & PySpark (Remote)

Binance; Townsville City; A distancia; AUD 150.000 - 210.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Learning budget

Performance bonuses

Wellbeing support

Remote-first

Descripción de la vacante

Halo Labs is hiring a Senior Data Warehouse Engineer to design and operate scalable data pipelines in a cloud-first SaaS environment. You will build PySpark-based frameworks, integrate data from multiple sources, and ensure production-grade reliability.

The role emphasizes collaboration with software engineers, data modeling, and secure data integration across Azure and AWS platforms.

Formación

• Strong hands-on experience with Azure Databricks and PySpark.
• Experience in designing and maintaining scalable ETL/ELT pipelines.
• Proficiency with cloud data architectures, data lakes, and governance.

Responsabilidades

• Design, develop, and maintain reusable PySpark frameworks for the SaaS platform.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing solutions using Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating clinical systems and external platforms.
• Design secure data integration patterns with REST APIs, SFTP, and webhooks.
• Collaborate with software engineers to embed data services into the product architecture.
• Contribute to backend design for data layer scalability, observability, and performance.
• Implement CI/CD pipelines with Azure DevOps, Git, and Azure Pipelines.
• Apply IaC and environment management across Azure and AWS.
• Optimize Spark jobs, clusters, and storage for performance and cost.
• Design and maintain data models including dimensional models and SaaS schemas.
• Implement data validation, monitoring, and alerting for production pipelines.
• Provide mentoring and enforce engineering standards across analytics and data teams.

Conocimientos

Python

PySpark

SQL

Databricks

Azure

AWS

Data Modeling

DevOps / CI-CD

Educación

Bachelor's degree in Computer Science, Engineering, Data Science, or related technical field

Herramientas

Azure Databricks

AWS S3

Git

Azure DevOps

REST APIs

CI/CD pipelines

Descripción del empleo

Halo Labs is hiring a Senior Data Warehouse Engineer to design and operate scalable data pipelines in a cloud-first SaaS environment. You will build PySpark-based frameworks, integrate data from multiple sources, and ensure production-grade reliability.

The role emphasizes collaboration with software engineers, data modeling, and secure data integration across Azure and AWS platforms.