Senior Data Engineer (Dev Background)

Nuage; Melbourne; On-site; AUD 90,000 - 130,000; Full time

Job summary

Nuage, a leading Australian brand, is building the streaming side of a major platform program in Melbourne. We are seeking a data engineer who can ensure real-time data behaves under changing platform conditions.

You will build and run streaming pipelines on Databricks and AWS, from ingestion through to trusted analytics-ready data, with a focus on latency, reliability, and cost efficiency.

Qualifications

• Strong hands-on Databricks streaming experience in production.
• Solid Python coding experience.

Responsibilities

• Build and own streaming pipelines using Databricks Structured Streaming, Auto Loader and Delta Lake.
• Ingest from Kafka (MSK) and Kinesis into a Bronze, Silver, Gold lakehouse.
• Keep data correct: dedup, late and out-of-order events, schema changes and safe replays.
• Monitor and tune pipelines for latency, cost and reliability.

Skills

Databricks streaming

Python

Kafka / Kinesis

AWS data services

CI/CD / IaC

Tools

Delta Lake

MSK

S3

Glue

Lambda

Athena

EMR

CloudWatch

Job description

A leading Australian brand is building out the streaming side of a major platform program, and they need a data engineer who can make real time data behave. Think high volume pipelines that have to stay right while the platform around them changes. You'll build and run streaming pipelines on Databricks and AWS, from ingestion through to trusted, analytics ready data.

What you'll do

• Build and own streaming pipelines using Databricks Structured Streaming, Auto Loader and Delta Lake
• Ingest from Kafka (MSK) and Kinesis into a Bronze, Silver, Gold lakehouse
• Keep data correct: dedup, late and out-of-order events, schema changes and safe replays
• Monitor and tune pipelines for latency, cost and reliability

What you bring

• Strong hands-on Databricks streaming experience in production, not just batch
• Solid Python coding experience (software)
• Real experience with Kafka or Kinesis
• AWS data services such as S3, Glue, Lambda, Athena, EMR and CloudWatch
• Comfort in CI/CD and infrastructure as code (Terraform or CDK)

The details

• Type: 6 month contract with high multiple extension possibility
• Location: Melbourne CBD