Data Engineer - AWS, Pyspark, Databricks

Profusion Group; City of Melbourne; On-site; AUD 120,000 - 160,000; Full time

Job summary

Profusion Group is seeking a Data Engineer for a 12-month contract with a major enterprise organisation. You will own data assets from ingestion through transformation, reconciliation, and production delivery, in a cloud lakehouse migration using AWS, PySpark, and Databricks.

Join a small, high-performing team in Melbourne and help modernise enterprise data platforms, delivering high-quality data solutions to critical business functions.

Qualifications

• Extensive data engineering experience in enterprise-scale environments.
• Advanced Python and SQL development skills.
• Hands-on Spark/PySpark and Databricks experience.
• Experience with AWS data services (Glue, Athena, S3, Lambda, Step Functions).
• Familiarity with CI/CD and IaC practices.

Responsibilities

• Design and build scalable AWS-based data pipelines with PySpark.
• Drive data quality, reconciliation, and platform reliability.
• Lead data transformations and production delivery in a lakehouse migration.
• Collaborate with business and tech stakeholders to deliver trusted data outcomes.
• Support Databricks migration initiatives.

Skills

Python

SQL

Spark / PySpark

AWS data services

Data modelling

Tools

Databricks

Delta Lake

CI/CD

Git

Infrastructure as Code

Job description

• 12-month contract with a major enterprise organisation
• End-to-end ownership across cloud data engineering and data transformation initiatives
• Work with AWS, PySpark, Databricks and modern lakehouse technologies

• 12-month contract with a major enterprise organisation
• End-to-end ownership across cloud data engineering and data transformation initiatives
• Work with AWS, PySpark, Databricks and modern lakehouse technologies

Company Overview

Our client is a highly respected global organisation undertaking a significant data transformation program.

Role Overview

As a Data Engineer, you'll play a key role in modernising enterprise data platforms and delivering high-quality data solutions that support critical business functions.

Working within a small, high-performing team, you'll own data assets from ingestion through to transformation, reconciliation, and production delivery.

This is an opportunity to work as part of a major cloud lakehouse migration, helping reshape how data is managed, consumed, and trusted across the organisation.

Key Responsibilities

• Design and build scalable AWS and PySpark data pipelines.
• Drive data quality, reconciliation, and platform reliability.
• Solve complex data challenges and optimise large-scale transformations.
• Support a major cloud lakehouse and Databricks migration initiative.
• Partner with business and technology stakeholders to deliver trusted data outcomes

Skills & Experience

We are seeking experienced Data Engineers who bring:

• Strong commercial experience in Data Engineering, including experience working within complex enterprise environments.
• Advanced Python and SQL development skills.
• Strong hands-on experience with Spark and PySpark.
• Experience working with AWS data services such as Glue, Athena, S3, Lambda and Step Functions.
• Databricks / Delta Lake experience highly regarded but not mandatory.
• Strong data modelling, data quality and reconciliation expertise.
• Experience with CI/CD practices, Git-based development and Infrastructure as Code principles.
• The ability to work autonomously, solve complex problems and take ownership of outcomes.

What's on Offer

People enjoy working in this team because they are trusted to solve challenging problems, own meaningful outcomes, and contribute to a large-scale transformation program that genuinely impacts the organisation's future direction.

Profusion respect people, value diversity and are committed to equality. We are committed to providing a supportive culture and positively contributing towards creating diverse and inclusive workplaces for our candidates & clients. We invite candidates of all ages, genders, sexual orientation, cultural backgrounds, people with disability, neurodiverse individuals, and Indigenous Australians to apply.