Cloud-Native Data Engineer: AWS, Spark & Pipelines

Umbrella Club; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Umbrella Club is seeking an experienced Data Engineer to join a team focused on building and supporting scalable, cloud-native data platforms in Sydney. You will design, develop and support data pipelines using AWS services and Spark/PySpark, with a strong emphasis on data modelling, governance and CI/CD practices.

The role requires hands-on SQL and Python, experience with CDC and operational databases, and collaboration with stakeholders in an Agile delivery environment.

Qualifications

• Design, develop and support cloud-native data platforms and pipelines.
• Hands-on experience with SQL and Python; strong AWS knowledge (S3, Glue, EMR, Athena, Aurora PostgreSQL).
• Experience with CDC, data governance, CI/CD and monitoring; Agile delivery environment.

Responsibilities

• Design, develop and support cloud-native data platforms and data pipelines.
• Build and maintain ETL/ELT pipelines using AWS services.
• Work with SQL, Python and Spark/PySpark for data processing.
• Develop with AWS S3, Glue, EMR, Athena and Aurora PostgreSQL.
• Apply data modelling and database design principles for scalable solutions.
• Work with CDC and operational databases.
• Support data governance, monitoring and CI/CD processes.
• Collaborate with stakeholders and delivery teams in Agile environments.
• Troubleshoot data pipeline and platform issues and contribute to improvements.

Skills

SQL

Python

Spark/PySpark

Agile delivery

Stakeholder engagement

Problem-solving

Tools

AWS S3

AWS Glue

AWS EMR

AWS Athena

Aurora PostgreSQL

GitHub Actions

Terraform

dbt

Snowflake

Job description

Umbrella Club is seeking an experienced Data Engineer to join a team focused on building and supporting scalable, cloud-native data platforms in Sydney. You will design, develop and support data pipelines using AWS services and Spark/PySpark, with a strong emphasis on data modelling, governance and CI/CD practices.

The role requires hands-on SQL and Python, experience with CDC and operational databases, and collaboration with stakeholders in an Agile delivery environment.