Flexible Real-Time Data Engineer | Kinesis & Databricks

Accenture Australia; Canberra; On-site; AUD 120,000 - 180,000; Full time

Benefits offered by this job

18 weeks parental leave

Career development program

Flexible work arrangements

Diversity & Inclusion

Job summary

Accenture Australia in Canberra seeks a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will work with AWS data technologies to move data from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will develop streaming solutions, ingest data into Delta tables, optimise pipelines for performance, and write clean Python and SQL while collaborating with batch data engineers.

Qualifications

• Hands-on production experience with Amazon Kinesis Data Streams.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on Databricks experience, including Delta Lake is a plus.

Responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Collaborate with batch data engineers to integrate streaming and batch pipelines.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Skills

Kinesis Data Streams

Python

SQL

Spark Structured Streaming

Databricks

Kafka

CloudWatch

Tools

Delta Lake

Job description

Accenture Australia in Canberra seeks a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will work with AWS data technologies to move data from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will develop streaming solutions, ingest data into Delta tables, optimise pipelines for performance, and write clean Python and SQL while collaborating with batch data engineers.