Real-Time Data Engineer: Kinesis, Databricks & Spark

Accenture Global; Canberra; Vor Ort; AUD 120.000 - 160.000; Vollzeit

Benefits dieser Stelle

18 weeks paid parental leave

Career development program

Flexible Work Arrangements

Family Inclusive Workplace

Diversity & Inclusion recognition

Zusammenfassung

Accenture is seeking a hands-on Data Engineer in Canberra to build reliable, real-time data pipelines using Amazon Kinesis and Databricks. You will move data from Kinesis into a Databricks Lakehouse with Spark Structured Streaming and Delta tables, optimizing for performance, latency, and cost.

You’ll collaborate with batch data engineers, write clean Python and SQL, and implement monitoring with CloudWatch. Security clearance for Australian citizenship is required.

Qualifikationen

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on experience with Databricks, Delta Lake and Unity Catalog.
• Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.

Aufgaben

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to integrate streaming and batch pipelines.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and drive improvements.

Kenntnisse

Python

SQL

Spark Structured Streaming

Troubleshooting

Kafka

Tools

Amazon Kinesis

Kinesis Data Firehose

Databricks

Delta Lake

Unity Catalog

IAM

S3

CloudWatch

Apache Spark

Jobbeschreibung

Accenture is seeking a hands-on Data Engineer in Canberra to build reliable, real-time data pipelines using Amazon Kinesis and Databricks. You will move data from Kinesis into a Databricks Lakehouse with Spark Structured Streaming and Delta tables, optimizing for performance, latency, and cost.

You’ll collaborate with batch data engineers, write clean Python and SQL, and implement monitoring with CloudWatch. Security clearance for Australian citizenship is required.