Remote Data Engineering & ETL/PySpark Architect

Mindrift; Warrnambool; Remote; AUD 120,000 - 160,000; Full time

Benefits offered by this job

Learning budget

Performance bonuses

Wellbeing support

Job summary

You will work with engineers, implement CI/CD, manage Databricks clusters, and apply data modelling best practices while upholding governance and security standards in regulated environments.

Qualifications

• Hands-on experience with Azure services: Databricks, Data Factory, SQL, Storage, and DevOps.
• Proficiency in Python, SQL, and PySpark for large-scale distributed data processing.
• Experience building production-grade data pipelines in SaaS environments; cloud-native architecture knowledge.

Responsibilities

• Design, develop, and maintain reusable in-house PySpark frameworks and pipelines.
• Architect and implement scalable ETL/ELT pipelines across Azure and AWS.
• Develop batch and near real-time ingestion workflows integrating external systems and APIs.
• Design secure data integration patterns and ensure data integrity and compliance.
• Collaborate with software engineers to embed data services into the SaaS product.

Skills

Azure Databricks

Python

SQL

PySpark

ETL/ELT pipelines

Cloud architecture

APIs integration

Data governance

DevOps

Data modeling

CI/CD

Education

Degree in Computer Science/Engineering/Data Science

Tools

Databricks

Azure DevOps

Azure Data Factory

Job description

You will work with engineers, implement CI/CD, manage Databricks clusters, and apply data modelling best practices while upholding governance and security standards in regulated environments.