Sr. Data Engineer

Showtime Consulting; Sydney; On-site; AUD 120,000 - 170,000; Full time

Job summary

Showtime Consulting is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

Responsibilities include delivering scalable ETL/ELT pipelines, supporting real-time integrations, and implementing Medallion Architecture with Delta Lake, while collaborating with data, engineering and business stakeholders across cloud platforms such as

Qualifications

• Hands-on data engineering with Azure Databricks and Spark (Scala).
• Strong SQL and Python for data processing.
• Experience delivering ETL/ELT pipelines and data warehousing concepts.
• Familiarity with Medallion Architecture and Delta Lake.
• Experience with CI/CD and observability in data platforms.
• Excellent communication and stakeholder engagement.

Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

Skills

Azure Databricks

Spark (Scala)

SQL

Python

ETL/ELT design

Medallion Architecture

Delta Lake

CI/CD

Observability

Stakeholder management

Tools

GitHub

Jenkins

Confluence

Jira

Azure DevOps

Job description

Build scalable data pipelines using Azure Databricks, Spark, Scala, SQL and Python within a large enterprise data environment.

16th October, 2026

Company Description

Showtime Consulting is a leading provider of Shielded Cloud and Digital Solutions across Australia and New Zealand. We specialise in delivering secure, enterprise-scale technology solutions across cloud, infrastructure, cybersecurity, business applications, and digital transformation programs.

We partner with government and enterprise organisations to build high-performing technology teams that deliver scalable, efficient, and future-ready business solutions.

The Role

We are seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney.

This is a hands-on engineering role focused on designing, building, and optimising high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will be responsible for delivering scalable ETL/ELT pipelines, supporting real-time and near-real-time integrations, and developing modern data solutions using Medallion Architecture and Delta Lake best practices.

Working closely with data, engineering and business stakeholders, you will also help operationalise workloads through CI/CD, scheduling, secrets management and observability.

Key Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

16th October, 2026

Company Description

Showtime Consulting is a leading provider of Shielded Cloud and Digital Solutions across Australia and New Zealand. We specialise in delivering secure, enterprise-scale technology solutions across cloud, infrastructure, cybersecurity, business applications, and digital transformation programs.

We partner with government and enterprise organisations to build high-performing technology teams that deliver scalable, efficient, and future-ready business solutions.

The Role

We are seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney.

This is a hands-on engineering role focused on designing, building, and optimising high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will be responsible for delivering scalable ETL/ELT pipelines, supporting real-time and near-real-time integrations, and developing modern data solutions using Medallion Architecture and Delta Lake best practices.

Working closely with data, engineering and business stakeholders, you will also help operationalise workloads through CI/CD, scheduling, secrets management and observability.

Key Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

What You’ll Need

• Strong hands-on experience with Azure Databricks.
• Strong commercial experience with Spark using Scala.
• Strong SQL skills.
• Strong Python skills.
• Experience designing and building scalable ETL/ELT data pipelines.
• Strong understanding of modern data warehousing concepts.
• Experience with Medallion Architecture and Delta Lake.
• Proven experience designing real-time and near-real-time integrations.
• Experience with cloud platforms, particularly Azure and/or AWS.
• Experience with big data technologies such as Hive, Impala, Sqoop and Spark.
• Experience with real-time data processing frameworks such as Spark Streaming and Apache/Confluent Kafka.
• Understanding of CI/CD practices and associated tools.
• Excellent communication and stakeholder management skills.

Preferred Skills

• Experience working with both Azure and AWS environments.
• Experience with Master Data Management solutions.
• Exposure to large-scale enterprise data platforms.
• Experience with tools including GitHub, Jenkins, Jira and Confluence.
• Strong understanding of data warehousing concepts.
• Experience operationalising Databricks workloads.
• Understanding of automation, monitoring and observability within data environments.