Senior Backend Engineer - Big Data & PySpark (Remote)

Binance; Townsville City; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Remote-first with travel to client loc

Dedicated learning budget

Job summary

Halo Labs is seeking a data engineer to design, develop, and maintain PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS. You will build distributed data processing on Databricks, ingest data in batch and near real time, and ensure secure, compliant data integration with third-party systems.

You will collaborate with software engineers to embed data services in the SaaS product, optimize performance, and implement CI/CD and IaC practices across Azure and AWS.

Qualifications

• Must have hands-on data engineering experience using Python, SQL and PySpark.
• Experience building scalable ETL/ELT pipelines in cloud environments.
• Knowledge of data modelling, governance and best practices for SaaS platforms.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks for standardized data engineering patterns.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS.
• Build distributed data processing with Python and PySpark on Databricks.
• Develop batch and near real-time ingestion pipelines integrating third-party systems.
• Design secure data integration patterns (APIs, SFTP, webhooks) ensuring data integrity.
• Collaborate with software engineers to embed data services into the SaaS product.
• Contribute to backend design for scalable data layers and observability.
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines.
• Apply IaC and environment management across Azure and AWS.
• Optimize Spark jobs and storage for performance and cost.
• Maintain robust data models and data schemas for SaaS.
• Implement data validation, monitoring, and alerting for reliability.
• Provide mentorship and enforce engineering standards.

Skills

Python

SQL

PySpark

Cloud architecture

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or related technical field

Tools

Azure Databricks

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

AWS

Databricks

S3

Lambda

Glue

Job description

Halo Labs is seeking a data engineer to design, develop, and maintain PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS. You will build distributed data processing on Databricks, ingest data in batch and near real time, and ensure secure, compliant data integration with third-party systems.

You will collaborate with software engineers to embed data services in the SaaS product, optimize performance, and implement CI/CD and IaC practices across Azure and AWS.