Senior Data Engineer — AWS Data Pipelines & ML-Ready Data

DFP Recruitment; New South Wales; On-site; AUD 150,000 - 210,000; Full time

Job summary

DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain reliable data pipelines and data products enabling data science, ML and AI initiatives. You will work with data scientists, solution architects and platform teams to translate complex data requirements into production-ready datasets and services.

You will apply software engineering, cloud and DevOps practices to ensure data quality, lineage, security and monitoring, and you'll optimize SQL, Python and cloud

Qualifications

• Production-ready data pipelines and data products in AWS.
• Strong ETL/ELT, data modelling and data quality controls.
• Experience with Python and modern DevOps practices.
• Ability to translate business requirements into technical designs.

Responsibilities

• Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases.
• Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources.
• Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines.
• Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria.
• Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting.
• Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency.
• Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions.
• Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions.
• Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements.

Skills

Python

Advanced SQL

ETL/ELT

Data modelling

Orchestration

CI/CD

Infrastructure-as-code

Docker

AWS

Security controls

Monitoring

Tools

Snowflake

Databricks

Redshift

Spark

dbt

Job description

DFP Recruitment is seeking a Senior Data Engineer to design, build and maintain reliable data pipelines and data products enabling data science, ML and AI initiatives. You will work with data scientists, solution architects and platform teams to translate complex data requirements into production-ready datasets and services.

You will apply software engineering, cloud and DevOps practices to ensure data quality, lineage, security and monitoring, and you'll optimize SQL, Python and cloud