Real-Time Data Engineer: AWS Kinesis & Databricks

Accenture PLC; Melbourne; On-site; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Parental leave (18 wks)

Career development

Flexible work arrangements

Global opportunities

Diversity & Inclusion rank

Job summary

Accenture is seeking a hands-on Data Engineer in Melbourne to build reliable, real-time data pipelines using Amazon Kinesis and Databricks. You will move data from Kinesis to the Databricks Lakehouse via Spark Structured Streaming and Delta tables, optimizing for performance, latency, and cost.

You will ingest streaming data into Delta tables, develop Python and SQL code, and collaborate with batch engineers to ensure seamless integration of streaming and batch workloads.

Qualifications

• Hands-on experience with streaming data pipelines.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on experience with Databricks, Delta Lake and Unity Catalog.
• Good knowledge of AWS services: IAM, S3, CloudWatch.

Responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to integrate streaming and batch pipelines.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Skills

Python

SQL

Spark Structured Streaming

Kinesis

Databricks

Delta Lake

Unity Catalog

CloudWatch

IAM

S3

Kafka

Tools

Databricks

Delta Lake

Kinesis Data Streams

Kinesis Data Firehose

Spark

AWS CloudWatch

Job description

Accenture is seeking a hands-on Data Engineer in Melbourne to build reliable, real-time data pipelines using Amazon Kinesis and Databricks. You will move data from Kinesis to the Databricks Lakehouse via Spark Structured Streaming and Delta tables, optimizing for performance, latency, and cost.

You will ingest streaming data into Delta tables, develop Python and SQL code, and collaborate with batch engineers to ensure seamless integration of streaming and batch workloads.