Data Engineer - Azure Databricks

CareCone Group; Sydney; On-site; AUD 120,000 - 150,000; Full time

Job summary

CareCone Group seeks a hands-on Developer Programmer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. You will design scalable ETL/ELT processes, implement real-time data integrations, and operationalize workloads with CI/CD and observability.

Proficiency in Delta Lake and medallion data architecture is a plus. The role emphasizes end-to-end pipeline design, performance tuning, and collaboration with data stakeholders.

Qualifications

• Hands-on role to design, build and optimize data pipelines on Azure Databricks using Spark & SQL.

Responsibilities

• Design and develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL.
• Design real-time and near real-time integrations.
• Operationalize workloads: schedule jobs/workflows, CI/CD with Git (Azure DevOps), secrets management and observability.

Skills

Databricks

SQL

Python

Spark (Scala)

Data warehousing concepts

Tools

Azure Databricks

Delta Lake

Job description

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

We‿re looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL

• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.