AWS Data Engineering Lead

The Recruitment Company; Sydney; On-site; AUD 140,000 - 180,000; Full time

Job summary

The Recruitment Company is seeking Seniors AWS Data Engineers in Sydney to join a high-profile data transformation programme. You will design and deliver production-grade data pipelines, build scalable data platforms, and work with modern AWS, Databricks and AI-enabled workflows.

You'll collaborate with a talented team of data and engineering specialists to underpin the organisation's analytics and AI capabilities, shaping the next generation of its data ecosystem in a non-BAU environment.

Qualifications

• Proven experience delivering production-grade data pipelines.
• Strong Python, SQL and Spark skills.
• Experience with Databricks, Delta Lake and Unity Catalog.
• Knowledge of modern cloud data platforms and lakehouse architectures.
• Experience with CI/CD and infrastructure as code (IaC).
• Financial services experience will be highly regarded.

Responsibilities

• Design, build and optimize data pipelines across the engineering lifecycle.
• Develop scalable data platforms that underpin analytics and AI capabilities.
• Collaborate with data and engineering teams to implement best practices.
• Contribute to data foundations supporting advanced analytics and AI initiatives.

Skills

AWS Data Engineering

Python

SQL

Spark / PySpark

CI/CD / DevOps mindset

Tools

Databricks

Delta Lake

Unity Catalog

Airflow / MWAA

Terraform

AWS CDK

Job description

We're partnering with an award-winning organisation that is making a significant investment in its data, cloud and AI capabilities.

With a major transformation programme commencing in Jan 2027 this is an exciting opportunity to join a high-profile initiative, working alongside a talented team of data and engineering specialists to build modern, scalable data platforms that will underpin the organisation's future analytics and AI capabilities.

This is not a BAU environment. You'll be working with modern technologies, solving complex engineering challenges and helping shape the next generation of the organisation's data ecosystem.

We're partnering with an award-winning organisation that is making a significant investment in its data, cloud and AI capabilities.

With a major transformation programme commencing in Jan 2027 this is an exciting opportunity to join a high-profile initiative, working alongside a talented team of data and engineering specialists to build modern, scalable data platforms that will underpin the organisation's future analytics and AI capabilities.

This is not a BAU environment. You'll be working with modern technologies, solving complex engineering challenges and helping shape the next generation of the organisation's data ecosystem.

The Opportunity

We're looking for strong Senior AWS Data Engineers who enjoy getting hands-on with technology and delivering high-quality data solutions into production.

You’ll be involved across the engineering lifecycle, from designing and building data pipelines through to optimising enterprise-scale platforms and establishing robust engineering practices.

What You’ll Be Working With

• AWS: S3, Glue, Lake Formation, Redshift, Athena and SageMaker.
• Databricks: Unity Catalog, Delta Lake and Spark.
• Development: Python, SQL and PySpark.
• Data Engineering: Modern ETL/ELT frameworks, data lakehouse architectures and data modelling.
• Streaming: Kafka, AWS MSK and Flink.
• Orchestration: Airflow / MWAA.
• DevOps: Terraform, AWS CDK and CI/CD pipelines.

You’ll also be contributing to data foundations that support advanced analytics, machine learning and emerging AI initiatives, including opportunities to explore AWS Bedrock and AI-enabled workflows.

What We’re Looking For

• Strong commercial experience in AWS Data Engineering.
• Proven experience designing, building and delivering production-grade data pipelines.
• Strong Python, SQL and Spark / PySpark skills.
• Experience with Databricks, ideally Delta Lake and Unity Catalog.
• Knowledge of modern cloud data platforms and lakehouse architectures.
• Experience with CI/CD, infrastructure as code and automated deployments.
• A strong engineering mindset, with a passion for modern technology and continuous improvement.
• Financial services experience will be highly regarded.

Why This Opportunity?

• Join a major, high-profile technology transformation programme.
• Be part of a significant investment in modern data and AI capabilities.
• Work with the latest AWS and Databricks technologies.
• Collaborate with a talented team of data and engineering specialists.
• Build platforms that will support the organisation's future analytics and AI ambitions.
• Opportunity to join on either a permanent or daily-rate contract basis.

If you're a strong AWS Data Engineer ready to take ownership of your next challenge, this is an opportunity to work on a high-impact programme where modern data engineering, cloud technology and AI innovation are firmly on the agenda.

I’d love to hear from you!