Senior Data Engineer: Secure Cloud Data Pipelines

Showtime Consulting Pty Ltd; Canberra; On-site; AUD 110,000 - 160,000; Full time

Job summary

Showtime Consulting Pty Ltd is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python, delivering scalable ETL/ELT solutions while applying Medallion Architecture and Delta Lake best practices.

Collaborate with stakeholders across data, engineering and delivery to enable real-time insights.

Qualifications

• Hands-on experience with Azure Databricks.
• Commercial experience with Spark (Scala).
• Strong SQL skills.
• Strong Python skills.
• Experience designing scalable ETL/ELT data pipelines.
• Strong understanding of modern data warehousing concepts.
• Experience with Medallion Architecture and Delta Lake.
• Proven experience designing real-time and near-real-time integrations.
• Experience with cloud platforms, particularly Azure and/or AWS.
• Experience with big data technologies such as Hive, Impala, Sqoop and Spark.
• Experience with real-time data processing frameworks such as Spark Streaming and Apache/Confluent Kafka.
• Understanding of CI/CD practices and associated tools.
• Excellent communication and stakeholder management skills.

Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

Skills

Azure Databricks

Spark (Scala)

SQL

Python

ETL/ELT pipelines

Medallion Architecture

Delta Lake

CI/CD

Observability

Cloud platforms (Azure/AWS)

Tools

Git

Azure DevOps

GitHub

Jenkins

Jira

Confluence

Databricks

Job description

Showtime Consulting Pty Ltd is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python, delivering scalable ETL/ELT solutions while applying Medallion Architecture and Delta Lake best practices.

Collaborate with stakeholders across data, engineering and delivery to enable real-time insights.