AWS Data Engineer/Data SME

AUSDX PTY LTD; Sydney; On-site; AUD 140,000 - 200,000; Full time

Job summary

AUSDX PTY LTD in Sydney is seeking a Senior AWS Data Engineer to design, build, and operate scalable data pipelines on AWS. The role focuses on ingesting, transforming, and serving data for reporting, analytics, and downstream applications, with hands-on SQL/Python and AWS data services.

You will collaborate with cross-functional teams, implement robust DataOps/DevOps practices, and contribute to data quality, governance, and performance tuning across batch and near real-time pipelines.

Qualifications

• 8–10 years of experience as a Data Engineer.
• Strong SQL skills and Python/PySpark experience.
• Hands-on with AWS data services (S3, Glue, Redshift, Athena, EMR).
• Experience with data warehousing concepts and data modeling.
• Experience with dbt and Airflow for orchestration.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines on AWS.
• Build and manage ETL/ELT workflows using AWS tools and dbt.
• Ingest data from diverse sources into lake/warehouse layers.
• Develop transformation logic with SQL and Python/PySpark.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with analysts/scientists for analytics-ready datasets.
• Contribute to DataOps/DevOps practices and produce runbooks.
• Optimize pipeline performance and support production operations.

Skills

Advanced SQL

Python

PySpark

Spark

Data modeling

Tools

Teradata

Siebel CRM data sets

dbt

Airflow

AWS (S3, Glue, Redshift, Athena, EMR)

Job description

The Senior AWS Data Engineer is responsible for designing, building, and supporting scalable data pipelines and curated datasets on AWS. You will work with cross-functional teams to ingest, transform, and serve data for reporting, analytics, and downstream applications. The ideal candidate is hands-on, strong in SQL/Python, and experienced with AWS-native data services and modern data engineering practices.

Key responsibilities

Design, develop, and maintain end-to-end data pipelines (batch and near real-time) on AWS Data Platform

Build and manage ETL/ELT workflows using AWS services (e.g., AWS Glue, S3, Redshift, Athena, EMR), dbt and orchestration tools such as Airflow

Implement data ingestion patterns from diverse sources (databases, APIs, files, event streams) into lake/warehouse layers such as raw, cleansed, and curated data layers

Develop transformation logic using SQL and Python/PySpark for cleansing, enrichment, and standardisation

Implement robust data quality checks, reconciliation controls, and monitoring/alerting for failures and anomalies

Collaborate with data analysts/data scientists to model datasets for analytics and machine learning consumption

Contribute to DataOps/DevOps practices: version control, CI/CD, automated testing, release management, and operational support

Produce and maintain technical documentation (data flows, mappings, job schedules, runbooks, and operational procedures)

Optimise Data Pipeline performance and support workflow orchestration and scheduling

Support production deployments and operations

About you

8–10 years' experience as a Data Engineer

Advanced SQL skills

Hands-on experience working with Teradata and Siebel CRM data sets

Experience delivering data pipelines in a large-scale enterprise data platform environment

Strong hands-on AWS experience with common data services such as: Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR and dbt

Strong programming capability in Python and strong data transformation experience using PySpark (preferred) and/or Spark

Experience with workflow orchestration tools such as Airflow

Solid understanding of data warehousing concepts (dimensional modelling, partitioning, incremental loads, CDC concepts)

Experience implementing monitoring, logging, alerting, and operational support processes

Strong communication skills and ability to work with stakeholders to translate requirements into data deliverables