Senior Python & Spark Data Engineer (Remote)

Mindrift; Gladstone Regional; Remote; AUD 140,000 - 190,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Remote-first culture

Job summary

You will collaborate with software engineers to embed data services into the product, optimize Spark performance, and implement robust testing and CI/CD practices in a remote-first Australian environment.

Qualifications

• Hands-on experience with Azure Databricks and Azure Data Factory.
• Proficient in Python, SQL and PySpark for large-scale data processing.
• Experience building production data pipelines in a SaaS environment.

Responsibilities

• Design, develop, and maintain reusable PySpark frameworks across the SaaS platform.
• Architect and implement scalable ETL/ELT pipelines on Azure and AWS.
• Build distributed data processing using Python and PySpark on Databricks.
• Develop ingestion pipelines integrating healthcare APIs and external systems.
• Implement secure data integration patterns and CI/CD pipelines.

Skills

Python

PySpark

SQL

Spark optimization

Databricks

Education

Degree in Computer Science, Engineering, Data Science or related

Tools

Azure Databricks

Azure Data Factory

AWS (S3, Glue)

Job description

You will collaborate with software engineers to embed data services into the product, optimize Spark performance, and implement robust testing and CI/CD practices in a remote-first Australian environment.