Data Engineer - Azure Databricks

CareCone Group; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

CareCone Group is seeking a hands-on Data Engineer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. The role focuses on scalable ETL/ELT architecture within a modern data lake framework.

The ideal candidate will implement real-time and near real-time data integrations, ensuring robust pipelines with CI/CD and observability. Knowledge of Delta Lake and Azure DevOps is advantageous for efficient delivery.

Qualifications

• Hands-on developer with strong experience designing data pipelines.
• Proficient in building ETL/ELT pipelines on Spark/Scala and SQL.
• Experience with real-time or near real-time integrations.
• Familiar with CI/CD practices using Git and observability tooling.

Responsibilities

• Design, implement, and optimize scalable data pipelines on Azure Databricks.
• Develop and maintain ETL/ELT processes following Delta Lake best practices.
• Ensure real-time data integration and reliable data delivery.
• Operationalize workloads with scheduling, CI/CD, secrets, and observability.

Skills

Azure Databricks

Scala Spark

SQL

Python

Tools

Git

Azure DevOps

Delta Lake

Job description

Role : Data Engineer - Azure Databricks

Type of role - Permanent Position

Location : Sydney, Australia

Primary or Mandatory skills

• Azure Databricks
• Scala Spark
• SQL
• Python

Good to have skills

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

• We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.