Azure Databricks Data Engineer for Real-Time ETL & Streaming

CareCone Group

Sydney

On-site

AUD 120,000 - 160,000

Full time

3 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CareCone Group is seeking a hands-on Developer Programmer to design and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. The role focuses on scalable ETL/ELT development following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.

The ideal candidate will implement real-time and near real-time integrations, operationalize workloads with scheduling, CI/CD using Git, secrets management, and strong observability across cloud data

Qualifications

Hands-on developer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL.
Design & develop scalable ETL/ELT pipelines following medallion architecture (Bronze/Silver/Gold) and Delta Lake.
Proven expertise in real-time and near real-time integrations.
Operationalize workloads: jobs & workflows scheduling, CI/CD with Git, secrets management, and observability.
Expertise in cloud platforms and data platforms like Databricks.

Responsibilities

Design, build and optimize high-throughput data pipelines on Azure Databricks.
Develop scalable ETL/ELT pipelines using Spark (Scala) and SQL.
Implement real-time and near real-time data integrations.
Operationalize workloads with scheduling, CI/CD, and observability.

Skills

Azure Databricks

Spark (Scala)

SQL

Python

Kafka

Delta Lake

CI/CD (Git/Azure DevOps)

Observability

Tools

Git

Azure DevOps

Databricks

Spark Streaming

Delta Lake

Job description

The ideal candidate will implement real-time and near real-time integrations, operationalize workloads with scheduling, CI/CD using Git, secrets management, and strong observability across cloud data

Get your free, confidential resume review.

or drag and drop your file here.