Data Engineer – AWS Kinesis & Databricks - AU Citizen

Accenture PLC; Canberra; On-site; AUD 120,000 - 170,000; Full time

Benefits offered by this job

Parental leave (18 weeks)

Career breaks

Career development

International opportunities

Family Inclusive Workplace

Flexible work arrangements

Diversity focus

Job summary

Accenture PLC in Canberra is seeking a hands-on Data Engineer to design and operate real-time data pipelines using Amazon Kinesis and Databricks. You will build streaming pipelines from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will implement monitoring with AWS CloudWatch, optimize latency and cost, collaborate with batch data engineers, and write clean Python and SQL code, with a focus on security clearances.

Qualifications

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.

Responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Skills

Python

SQL

Apache Spark

AWS Kinesis

Databricks

Delta Lake

Unity Catalog

CloudWatch

Real-time data pipelines

Problem solving

Kafka

Tools

Kinesis Data Streams

Kinesis Data Firehose

Databricks

Spark Structured Streaming

Delta Lake

Unity Catalog

AWS IAM

AWS S3

AWS CloudWatch

Kafka

Job description

Data Engineer – AWS Kinesis & Databricks - AU Citizen

We’re looking for a hands‑on Data Engineer with strong experience building reliable, real‑time data pipelines using Amazon Kinesis and Databricks.

You’ll work with modern AWS data technologies to develop and optimise streaming pipelines, moving high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.

What you’ll do:

• Build and support real‑time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

What you’ll bring:

• Hands‑on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands‑on experience with Databricks, ideally including Delta Lake and Unity Catalog.
• Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
• Understanding of scalable and reliable data-pipeline design.
• Strong troubleshooting and problem-solving skills.
• Experience with Kafka is beneficial

Additional Requirements:
Security Clearance Requirement: Given the secure nature of this role, we are seeking candidates who are Australian Citizens with an active security clearance.

This requirement ensures that all team members are cleared to handle sensitive information and contribute to maintaining the highest standards of security in our operations.

Benefits of working at Accenture:

• 18 weeks paid parental leave
• Long & short-term career break opportunities
• Structured career development program
• Local and international career opportunities.
• Certified as a Family Inclusive Workplace
• Flexible Work Arrangements - centred around Accenture’s Truly Human ethos and our commitment to supporting the health and wellbeing of our people.
• We are proud to be in the top 3 of last year’s Diversity & Inclusion Index!

#LI_GM

Visit us atwww.accenture.com

About Accenture

Accenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services—creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. We combine our strength in technology and leadership in cloud, data and AI with unmatched industry experience, functional expertise and global delivery capability. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360 value, enable us to help our clients reinvent and build trusted, lasting relationships. We measure our success by the 360 value we create for our clients, each other, our shareholders, partners and communities.

Equal Employment Opportunity Statementfor Australia:

At Accenture, our intention is to foster a culture and a workplace in which all of our people feel a sense of belonging and are respected and empowered to do their best work.

As part of our talent strategy, we hire and develop people who have different backgrounds, different perspectives and different lived experiences. These differences ensure that we have and attract the cognitive diversity to deliver a variety of perspectives, observations and insights which are essential to drive the innovation needed to reinvent, and we hold our leaders accountable for ensuring we have the most innovative and talented people in our industry.

We encourage applications from all people, and we are committed to removing barriers to the recruitment process and employee lifecycle. All employment decisions shall be made without regard to age, disability status, ethnicity, gender, gender identity or expression, religion or sexual orientation and we do not tolerate discrimination. If you require adjustments to the recruitment process or have a preferred communication method, please email**********@accenture.comand cite the relevant Job Number, or contact us on+61 2 9005 5000.

Learn how Accenture protects your personal data and know your rights in relation to your personal data. Read more about ourPrivacy Statement