Real-Time Data Engineer: Kinesis & Databricks

Accenture Global; Melbourne; Vor Ort; AUD 120.000 - 170.000; Vollzeit

Benefits dieser Stelle

18 weeks paid parental leave

Career breaks (long & short-term)

Structured career development program

Local and international careeropportun

Family Inclusive Workplace

Flexible work arrangements

Diversity & Inclusion recognition

Zusammenfassung

Accenture Global in Melbourne is seeking a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will move high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.

You’ll collaborate with cloud data teams to ensure streaming and batch processes are well-integrated, write clean Python and SQL, and optimise pipelines for performance, latency and cost in a fast-paced environment.

Qualifikationen

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on experience with Databricks, ideally including Delta Lake and Unity Catalog.
• Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
• Understanding of scalable and reliable data-pipeline design.
• Strong troubleshooting and problem-solving skills.
• Experience with Kafka is beneficial.

Aufgaben

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Kenntnisse

Python

SQL

Spark Structured Streaming

Databricks

Kinesis

AWS

Troubleshooting

Kafka

Tools

Databricks

Spark

Kinesis

IAM

S3

CloudWatch

Jobbeschreibung

Accenture Global in Melbourne is seeking a hands-on Data Engineer to build reliable real-time data pipelines using Amazon Kinesis and Databricks. You will move high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.

You’ll collaborate with cloud data teams to ensure streaming and batch processes are well-integrated, write clean Python and SQL, and optimise pipelines for performance, latency and cost in a fast-paced environment.