Remote Senior Backend Engineer — Big Data & Spark

Binance; Coffs Harbour City Council; Remote; AUD 120,000 - 160,000; Full time

Benefits offered by this job

Learning budget

Performance bonus

Wellbeing support

Job summary

You will implement secure data integration patterns, optimise Spark workloads, and collaborate with software engineers to embed data services into the product architecture, while upholding data governance and reliability.

Qualifications

• Hands-on experience with Azure Databricks and Azure data tools.
• Proficient in Python, SQL, and PySpark for large-scale distributed processing.
• Deep experience configuring and managing Databricks clusters for scalable big data workloads.
• Experience building production-ready data pipelines in a SaaS or product-led environment.
• Understanding of data lakes and lakehouse architecture and modular pipeline design.
• Experience integrating with third-party systems via APIs and secure data exchange mechanisms.
• Exposure to healthcare or regulated data environments and handling data securely.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks to enforce standardised data engineering patterns across the SaaS platform
• Architect and implement scalable, production-grade ETL/ELT pipelines across Azure and AWS environments
• Build distributed data processing solutions using Python and PySpark on Databricks
• Develop batch and near real-time ingestion pipelines integrating third-party clinical systems, healthcare APIs, and external enterprise platforms
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion, webhooks) ensuring compliance and data integrity
• Work closely with Software Engineers to embed data services into the SaaS product architecture
• Contribute to backend system design discussions to ensure data layer scalability, observability, and performance
• Implement CI/CD pipelines using Azure DevOps, Git, and Azure Pipelines for automated deployment and testing of data workloads
• Apply infrastructure-as-code and environment management best practices across Azure and AWS
• Optimise Spark jobs, cluster configurations, and storage strategies for performance and cost efficiency
• Design and maintain robust data models, including dimensional models and SaaS-oriented data schemas
• Implement data validation, monitoring, and alerting to ensure pipeline reliability and production stability
• Provide technical mentorship and enforce engineering standards across the analytics and data engineering team

Skills

Python

SQL

PySpark

Databricks

Azure

AWS

Data pipelines

CI/CD

Education

Bachelor's degree in CS/Engineering/Data Science

Tools

Azure Databricks

Azure Data Factory

Azure SQL

Azure Storage

Azure DevOps

AWS S3

AWS Lambda

AWS Glue

Job description

You will implement secure data integration patterns, optimise Spark workloads, and collaborate with software engineers to embed data services into the product architecture, while upholding data governance and reliability.