Senior Python Data Engineer - Scraping & ETL (Remote)

Mindrift; Cairns Regional; Remote; AUD 83,000 - 165,000; Full time

Benefits offered by this job

Remote-first organisation

Learning budget

Performance bonuses

Wellbeing support

Job summary

You will work with Azure and AWS environments, optimise Spark jobs, implement CI/CD, and ensure data governance, security, and production stability while collaborating with software engineers.

This remote-first role requires strong Python, PySpark, Databricks expertise and ability to travel to client sites across Australia.

Qualifications

• Strong hands-on experience with Azure Databricks and Azure services
• Experience with AWS services (S3, Lambda, RDS, Glue, IAM)
• Advanced Python, SQL, PySpark for large-scale processing
• Databricks cluster management for big data workloads
• Production-ready data pipelines in SaaS environments
• Cloud-native data architecture and lakehouse patterns
• API integrations and secure data exchange
• Healthcare or regulated data exposure
• Data modelling, metadata management, governance
• Automated testing frameworks for data pipelines
• DevOps with Git workflows and CI/CD
• Degree

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS
• Build distributed data processing solutions using Python and PySpark on Databricks
• Develop batch and near real-time ingestion pipelines integrating third-party systems
• Design secure data integration patterns (REST APIs, SFTP, event-driven ingestion)
• Collaborate with Software Engineers to embed data services into SaaS architecture
• Contribute to backend design discussions for data layer scalability
• Implement CI/CD pipelines for automated deployment and testing of data workloads
• Apply IaC and environment management across Azure and AWS
• Optimise Spark jobs, cluster configurations, and storage strategies
• Design robust data models and SaaS-oriented schemas
• Implement data validation, monitoring, and alerting for pipeline reliability
• Provide technical mentorship and enforce engineering standards across analytics team

Skills

Python

PySpark

SQL

Azure

AWS

DevOps

REST APIs

Data architecture

CI/CD

Healthcare data

Degree

Education

Degree

Tools

Databricks

Azure DevOps

Git

Azure Data Factory

Azure SQL

S3

REST APIs

Job description

You will work with Azure and AWS environments, optimise Spark jobs, implement CI/CD, and ensure data governance, security, and production stability while collaborating with software engineers.

This remote-first role requires strong Python, PySpark, Databricks expertise and ability to travel to client sites across Australia.