Staff Data Engineer - Payments Platforms Lead (AWS/Snowflake)

Commonwealth Bank; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

Commonwealth Bank in Australia is seeking a Staff Data Engineer for Payments Technology, based in Sydney or Melbourne. You will provide hands-on technical leadership for data platforms, data products, and pipelines that support Payments, solving complex problems at scale and influencing architecture across squads.

You will design and build batch and streaming pipelines using Spark, Airflow and AWS Glue, and develop cloud-native solutions on AWS services such as S3, EMR, Redshift and SageMaker.

Qualifications

• Proven experience building scalable data platforms and pipelines.
• Strong knowledge of AWS data services and data products.
• Experience with batch and streaming data processing.

Responsibilities

• Set technical direction for complex data engineering initiatives and influence architecture and standards across multiple squads.
• Design, build and evolve scalable batch and streaming data pipelines using Spark, Airflow and AWS Glue.
• Build cloud-native data solutions across AWS services (S3, EMR, Glue, Lambda, Redshift, SageMaker).
• Engineer high-quality data products and help evolve enterprise data platform and data marketplace patterns.
• Lead engineering practices across testing, CI/CD, infrastructure-as-code, observability and production reliability.
• Mentor engineers, support technical decision-making and raise data engineering capability across teams.
• Partner with Architecture, Product, Security and data consumers to translate complex requirements into pragmatic engineering solutions.

Skills

AWS data services

Python

PySpark

Spark

Airflow

Snowflake

SQL

CI/CD

Observability

Technical leadership

Tools

AWS Glue

S3

Redshift

EMR

SageMaker

Lambda

Job description

Commonwealth Bank in Australia is seeking a Staff Data Engineer for Payments Technology, based in Sydney or Melbourne. You will provide hands-on technical leadership for data platforms, data products, and pipelines that support Payments, solving complex problems at scale and influencing architecture across squads.

You will design and build batch and streaming pipelines using Spark, Airflow and AWS Glue, and develop cloud-native solutions on AWS services such as S3, EMR, Redshift and SageMaker.