Remote Data Scraping Engineer: PySpark & Pipelines

Mindrift; Toowoomba Regional; Remote; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Dedicated learning budget

Performance bonuses

Wellbeing support

Job summary

You will work in a remote-first environment with travel to client locations across Australia, contributing to architecture discussions and implementing CI/CD pipelines for automated deployment and testing of data workloads.

Qualifications

• Strong hands-on experience with Azure Databricks and Databricks platform
• Advanced proficiency in Python, PySpark, and SQL for large-scale data processing
• Experience building production-ready data pipelines and ETL/ELT workflows

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks across SaaS platform
• Architect scalable ETL/ELT pipelines in Azure and AWS
• Build distributed data processing with Python and PySpark on Databricks
• Develop batch and real-time ingestion pipelines from healthcare systems and external platforms
• Design secure data integration patterns (APIs, SFTP, webhooks) ensuring data integrity

Skills

Azure Databricks

Python

PySpark

SQL

APIs

CI/CD

Azure DevOps

Education

Degree in Computer Science, Engineering, Data Science, or related field

Tools

Databricks

Job description

You will work in a remote-first environment with travel to client locations across Australia, contributing to architecture discussions and implementing CI/CD pipelines for automated deployment and testing of data workloads.