Data Engineer — Hybrid, Real-Time Data Pipelines on AWS

ASX; Sydney; Hybrid; AUD 140,000 - 210,000; Full time

Job summary

ASX is powering Australia's financial markets. ASX invites you to join the Data & Integration Platform team to shape data strategy, leverage a central data platform, and deliver secure, scalable data products.

You will ingest core data assets, curate them into source-aligned products, and build dashboards, APIs and reports, while applying DevSecOps practices in a critical market infrastructure environment.

Qualifications

• Hands-on engineer with deep experience using AWS data services for ingestion, storage, orchestration, data management, analytics and distribution.
• Strong data modelling skills for structuring data within a data warehousing environment following best practice data management standards.
• Solid data product architecture with DevSecOps practices to automate complex data pipelines.
• Experience in real time streaming using Confluent Kafka and AWS services.
• Experience with Airflow, AWS Glue, Apache Iceberg, Athena, Redshift and DataZone.
• Experience in data visualization tools such as Tableau/Power BI/QuickSight.
• Solid experience in Python, Spark and Terraform/Terragrunt.

Responsibilities

• Embedded within one of our data and reporting teams, collaborate with stakeholders to understand data objectives and champion the central data platform.
• Work within a domain-aligned data product team with data owners and SMEs alongside Technology.
• Hands-on ingestion of ASX’s core data assets, curating them into source-aligned data products and creating consumable files, reports, dashboards and APIs.
• Contribute to a data engineering practice and drive standards, frameworks and best practices.
• Ensure data pipelines and products are designed to run within a critical market infrastructure environment by product teams.

Skills

AWS services

Data modelling

DevSecOps

Real-time streaming

Confluent Kafka

Python

Spark

Terraform/Terragrunt

Tools

Airflow

AWS Glue

Apache Iceberg

Athena

Redshift

Tableau

Power BI

QuickSight

DataZone

Job description

ASX is powering Australia's financial markets. ASX invites you to join the Data & Integration Platform team to shape data strategy, leverage a central data platform, and deliver secure, scalable data products.

You will ingest core data assets, curate them into source-aligned products, and build dashboards, APIs and reports, while applying DevSecOps practices in a critical market infrastructure environment.