Data Engineer – AWS Kinesis & Databricks - AU Citizen

Accenture Australia; Canberra; On-site; AUD 120,000 - 180,000; Full time

Benefits offered by this job

18 weeks parental leave

Career development program

Flexible work arrangements

Diversity & Inclusion

Job summary

Accenture Australia in Canberra seeks a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will work with AWS data technologies to move data from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will develop streaming solutions, ingest data into Delta tables, optimise pipelines for performance, and write clean Python and SQL while collaborating with batch data engineers.

Qualifications

• Hands-on production experience with Amazon Kinesis Data Streams.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on Databricks experience, including Delta Lake is a plus.

Responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Collaborate with batch data engineers to integrate streaming and batch pipelines.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Skills

Kinesis Data Streams

Python

SQL

Spark Structured Streaming

Databricks

Kafka

CloudWatch

Tools

Delta Lake

Job description

We're looking for a hands-on Data Engineer with strong experience building reliable, real-time data pipelines using Amazon Kinesis and Databricks. You'll work with modern AWS data technologies to develop and optimise streaming pipelines, moving high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.

What you'll do

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

What you'll bring

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on experience with Databricks, ideally including Delta Lake and Unity Catalog.
• Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
• Understanding of scalable and reliable data-pipeline design.
• Strong troubleshooting and problem-solving skills.
• Experience with Kafka is beneficial

Additional Requirements

Security Clearance Requirement

Given the secure nature of this role, we are seeking candidates who are Australian Citizens with an active security clearance. This requirement ensures that all team members are cleared to handle sensitive information and contribute to maintaining the highest standards of security in our operations.

Benefits of working at Accenture

• 18 weeks paid parental leave
• Long & short-term career break opportunities
• Structured career development program
• Local and international career opportunities.
• Certified as a Family Inclusive Workplace
• Flexible Work Arrangements - centred around Accenture's Truly Human ethos and our commitment to supporting the health and wellbeing of our people.
• We are proud to be in the top 3 of last year's Diversity & Inclusion Index!