Senior Data Engineer – Real-Time Pipelines on AWS

The Hr Ally; Sydney; On-site; AUD 150,000 - 210,000; Full time

Job summary

The Hr Ally is seeking an experienced Data Engineer (9-14 years) to join a high-performing team delivering enterprise-scale data platforms and real-time data solutions on AWS in Sydney.

You will design scalable batch and real-time data ingestion, optimize pipelines, and build low-latency streaming solutions using Kinesis or Kafka. The role demands strong AWS, Python, PySpark, and database expertise, with CI/CD and IaC know-how.

Qualifications

• Strong expertise in AWS, Python, PySpark, Airflow, and Redshift.
• Hands-on experience with SageMaker Unified Studio.
• Experience with relational and NoSQL databases (Oracle, MongoDB, DynamoDB, Snowflake, Teradata).
• Proven experience building real-time and batch data pipelines.
• Knowledge of CI/CD, Git, Infrastructure as Code (Terraform/CloudFormation).
• Data Modelling experience is highly desirable.

Responsibilities

• Design and develop scalable batch and real-time data ingestion frameworks.
• Build and optimize data pipelines using AWS services and modern data engineering practices.
• Develop low-latency streaming solutions using Kinesis, Kafka, or similar technologies.
• Implement data transformations, orchestration workflows, and monitoring solutions.
• Work with Data Warehouses, Data Lakes, and Lakehouse architectures.
• Collaborate with cross-functional teams to deliver robust and scalable data solutions.

Skills

Python

PySpark

AWS

Airflow

SQL

Tools

SageMaker Unified Studio

Kinesis

Kafka

Redshift

Snowflake

DynamoDB

MongoDB

Oracle

Teradata

Terraform/CloudFormation

Job description

The Hr Ally is seeking an experienced Data Engineer (9-14 years) to join a high-performing team delivering enterprise-scale data platforms and real-time data solutions on AWS in Sydney.

You will design scalable batch and real-time data ingestion, optimize pipelines, and build low-latency streaming solutions using Kinesis or Kafka. The role demands strong AWS, Python, PySpark, and database expertise, with CI/CD and IaC know-how.