Sr. Data Engineer

Showtime Consulting Pty Ltd; Sydney; Hybrid; AUD 120,000 - 180,000; Full time

Job summary

Showtime Consulting Pty Ltd in Sydney seeks an experienced Data Engineer to join a large enterprise data environment. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will deliver scalable ETL/ELT pipelines, support real-time integrations, and operationalise workloads through CI/CD, scheduling, secrets management and observability across cloud platforms such as Azure and AWS.

Qualifications

• Experience designing scalable ETL/ELT pipelines.
• Strong data warehousing concepts and Medallion Architecture.
• Hands-on with Azure Databricks and Spark.
• Proficiency in SQL and Python.
• Experience with CI/CD and cloud platforms.

Responsibilities

• Design and optimise high-throughput data pipelines.
• Develop ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and Azure DevOps.
• Implement secrets management and observability practices.
• Work across Azure and AWS, coordinating with stakeholders.

Skills

Communication

Stakeholder mgmt

Analytical thinking

Team collaboration

Tools

Azure Databricks

Spark (Scala)

SQL

Python

Git

Jenkins

Confluence

Azure DevOps

Delta Lake

Job description

Are you ready to be a part of a team that's redefining ultra-secure cloud solutions?

Join a team that’s shaping the future of cloud and cybersecurity.

At Showtime Consulting, we don’t just build ultra-secure cloud solutions—we enable organisations to thrive in the most challenging high-threat environments. If you’re truly passionate about innovation, security, and making a real impact, you’re in the right place.

We deliver security where it matters most—protecting national infrastructure, defence, and mission-critical operations. Our solutions are trusted by leading organisations, and our team is at the forefront of innovation in ultra-secure cloud and cross-domain security.

Our people are specialists in secure cloud, defence technology, and insider risk mitigation. We thrive in high-stakes environments, where cybersecurity isn’t just a requirement, it’s mission-critical. If you’re ready to contribute to industry-leading projects and grow in a company that values trust, innovation, and resilience, explore our open roles today.

Open Roles

We Are Always Looking For Talented Individuals.
Explore Our Current Opportunities And Find A Role That Matches Your Expertise.

Sydney

Company Description

Showtime Consulting is a leading provider of Shielded Cloud and Digital Solutions across Australia and New Zealand. We specialise in delivering secure, enterprise-scale technology solutions across cloud, infrastructure, cybersecurity, business applications, and digital transformation programs.

We partner with government and enterprise organisations to build high-performing technology teams that deliver scalable, efficient, and future-ready business solutions.

The Role

We are seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney.

This is a hands‑on engineering role focused on designing, building, and optimising high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will be responsible for delivering scalable ETL/ELT pipelines, supporting real-time and near-real-time integrations, and developing modern data solutions using Medallion Architecture and Delta Lake best practices.

Working closely with data, engineering and business stakeholders, you will also help operationalise workloads through CI/CD, scheduling, secrets management and observability.

Key Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

What You'll Need

• Strong hands‑on experience with Azure Databricks.
• Strong commercial experience with Spark using Scala.
• Strong SQL skills.
• Strong Python skills.
• Experience designing and building scalable ETL/ELT data pipelines.
• Strong understanding of modern data warehousing concepts.
• Experience with Medallion Architecture and Delta Lake.
• Proven experience designing real-time and near-real-time integrations.
• Experience with cloud platforms, particularly Azure and/or AWS.
• Experience with big data technologies such as Hive, Impala, Sqoop and Spark.
• Experience with real-time data processing frameworks such as Spark Streaming and Apache/Confluent Kafka.
• Understanding of CI/CD practices and associated tools.
• Excellent communication and stakeholder management skills.

Preferred Skills

• Experience working with both Azure and AWS environments.
• Experience with Master Data Management solutions.
• Exposure to large-scale enterprise data platforms.
• Experience with tools including GitHub, Jenkins, Jira and Confluence.
• Strong understanding of data warehousing concepts.
• Experience operationalising Databricks workloads.
• Understanding of automation, monitoring and observability within data environments.