Senior Data Engineer: Real-Time Pipelines on Databricks

Showtime Consulting; Sydney; Sur place; AUD 120 000 - 170 000; Plein temps

Résumé du poste

Showtime Consulting is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

Responsibilities include delivering scalable ETL/ELT pipelines, supporting real-time integrations, and implementing Medallion Architecture with Delta Lake, while collaborating with data, engineering and business stakeholders across cloud platforms such as

Qualifications

• Hands-on data engineering with Azure Databricks and Spark (Scala).
• Strong SQL and Python for data processing.
• Experience delivering ETL/ELT pipelines and data warehousing concepts.
• Familiarity with Medallion Architecture and Delta Lake.
• Experience with CI/CD and observability in data platforms.
• Excellent communication and stakeholder engagement.

Responsabilités

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

Connaissances

Azure Databricks

Spark (Scala)

SQL

Python

ETL/ELT design

Medallion Architecture

Delta Lake

CI/CD

Observability

Stakeholder management

Outils

GitHub

Jenkins

Confluence

Jira

Azure DevOps

Description du poste

Showtime Consulting is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

Responsibilities include delivering scalable ETL/ELT pipelines, supporting real-time integrations, and implementing Medallion Architecture with Delta Lake, while collaborating with data, engineering and business stakeholders across cloud platforms such as