Real-Time Data Engineer: AWS Kinesis & Databricks

Accenture PLC; Canberra; On-site; AUD 120,000 - 170,000; Full time

Benefits offered by this job

Parental leave (18 weeks)

Career breaks

Career development

International opportunities

Family Inclusive Workplace

Flexible work arrangements

Diversity focus

Job summary

Accenture PLC in Canberra is seeking a hands-on Data Engineer to design and operate real-time data pipelines using Amazon Kinesis and Databricks. You will build streaming pipelines from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will implement monitoring with AWS CloudWatch, optimize latency and cost, collaborate with batch data engineers, and write clean Python and SQL code, with a focus on security clearances.

Qualifications

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.

Responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Skills

Python

SQL

Apache Spark

AWS Kinesis

Databricks

Delta Lake

Unity Catalog

CloudWatch

Real-time data pipelines

Problem solving

Kafka

Tools

Kinesis Data Streams

Kinesis Data Firehose

Databricks

Spark Structured Streaming

Delta Lake

Unity Catalog

AWS IAM

AWS S3

AWS CloudWatch

Kafka

Job description

Accenture PLC in Canberra is seeking a hands-on Data Engineer to design and operate real-time data pipelines using Amazon Kinesis and Databricks. You will build streaming pipelines from Kinesis into the Databricks Lakehouse with Spark Structured Streaming and Delta tables.

You will implement monitoring with AWS CloudWatch, optimize latency and cost, collaborate with batch data engineers, and write clean Python and SQL code, with a focus on security clearances.