Senior Data Engineer - Scalable Data Pipelines & ML Ready

DFP Recruitment; Melbourne; On-site; AUD 140,000 - 190,000; Full time

Job summary

DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines and data products to support data science, ML, and analytics. You will work with data scientists, platform teams and stakeholders to translate requirements into production-ready datasets and services, applying cloud, DevOps and data governance practices.

The role focuses on AWS-based data engineering, performance, security and observability, with opportunities to influence architecture and

Qualifications

• Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services.
• Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data.
• Demonstrated application of modern software engineering and DevOps practices, including Git, code review, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation.
• Strong programming capability in Python, with experience developing maintainable, modular and testable production code. Experience with Java or Scala is also relevant.
• Advanced SQL skills, including query optimisation, performance tuning and working with large or complex datasets.
• Hands-on experience delivering cloud-based data solutions in containerised environments such as Docker, ECS or EKS, with a sound understanding of security, access controls, secrets management and operational monitoring.

Responsibilities

• Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases.
• Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources.
• Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines.
• Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria.
• Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting.
• Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency.
• Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions.
• Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions.
• Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements.

Skills

Python programming

Advanced SQL

ETL/ELT patterns

Data modelling

DevOps practices

CI/CD tooling

Tools

AWS S3

AWS Glue

AWS Lambda

AWS Athena

AWS EMR

AWS Kinesis

Terraform

AWS CloudFormation

Docker

ECS

EKS

Git

CI/CD

Job description

DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines and data products to support data science, ML, and analytics. You will work with data scientists, platform teams and stakeholders to translate requirements into production-ready datasets and services, applying cloud, DevOps and data governance practices.

The role focuses on AWS-based data engineering, performance, security and observability, with opportunities to influence architecture and