Data Engineer - Azure Databricks

CareCone Group; Sydney; On-site; AUD 140,000 - 190,000; Full time

Job summary

CareCone Group in Sydney is seeking a hands-on Developer/Programmer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. The role focuses on scalable ETL/ELT pipelines with medallion architecture and Delta Lake best practices.

You will implement real-time integrations, operationalize workloads, and manage CI/CD with Git and Azure DevOps, ensuring observability across data pipelines.

Qualifications

• Hands-on developer to design and optimize high-throughput data pipelines on Azure Databricks.
• Experience with Spark (Scala) and SQL in ETL/ELT pipelines.
• Implement medallion architecture (Bronze/Silver/Gold) and Delta Lake.
• Set up jobs, workflows, CI/CD with Git, secrets management, and observability.

Responsibilities

• Design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL.
• Develop scalable ETL/ELT pipelines following medallion architecture and Delta Lake.
• Design real-time and near real-time data integrations.
• Operationalize workloads: scheduling, CI/CD, secrets management, and observability.

Skills

Databricks

Spark (Scala)

SQL

Python

ETL/ELT pipelines

Medallion architecture

Delta Lake

CI/CD

Git

Observability

Real-time integrations

Tools

Azure Databricks

Delta Lake

Git

Azure DevOps

Job description

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

• We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.