Senior Data Engineer - Secure Cloud Data Pipelines (Azure)

Showtime Consulting Pty Ltd; Sydney; Hybrid; AUD 120,000 - 180,000; Full time

Job summary

Showtime Consulting Pty Ltd in Sydney seeks an experienced Data Engineer to join a large enterprise data environment. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will deliver scalable ETL/ELT pipelines, support real-time integrations, and operationalise workloads through CI/CD, scheduling, secrets management and observability across cloud platforms such as Azure and AWS.

Qualifications

• Experience designing scalable ETL/ELT pipelines.
• Strong data warehousing concepts and Medallion Architecture.
• Hands-on with Azure Databricks and Spark.
• Proficiency in SQL and Python.
• Experience with CI/CD and cloud platforms.

Responsibilities

• Design and optimise high-throughput data pipelines.
• Develop ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and Azure DevOps.
• Implement secrets management and observability practices.
• Work across Azure and AWS, coordinating with stakeholders.

Skills

Communication

Stakeholder mgmt

Analytical thinking

Team collaboration

Tools

Azure Databricks

Spark (Scala)

SQL

Python

Git

Jenkins

Confluence

Azure DevOps

Delta Lake

Job description

Showtime Consulting Pty Ltd in Sydney seeks an experienced Data Engineer to join a large enterprise data environment. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will deliver scalable ETL/ELT pipelines, support real-time integrations, and operationalise workloads through CI/CD, scheduling, secrets management and observability across cloud platforms such as Azure and AWS.