Azure Databricks Data Engineer: Real-Time ETL Pipelines

CareCone Group; Sydney; On-site; AUD 120,000 - 150,000; Full time

Job summary

CareCone Group seeks a hands-on Developer Programmer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. You will design scalable ETL/ELT processes, implement real-time data integrations, and operationalize workloads with CI/CD and observability.

Proficiency in Delta Lake and medallion data architecture is a plus. The role emphasizes end-to-end pipeline design, performance tuning, and collaboration with data stakeholders.

Qualifications

• Hands-on role to design, build and optimize data pipelines on Azure Databricks using Spark & SQL.

Responsibilities

• Design and develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL.
• Design real-time and near real-time integrations.
• Operationalize workloads: schedule jobs/workflows, CI/CD with Git (Azure DevOps), secrets management and observability.

Skills

Databricks

SQL

Python

Spark (Scala)

Data warehousing concepts

Tools

Azure Databricks

Delta Lake

Job description

CareCone Group seeks a hands-on Developer Programmer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. You will design scalable ETL/ELT processes, implement real-time data integrations, and operationalize workloads with CI/CD and observability.

Proficiency in Delta Lake and medallion data architecture is a plus. The role emphasizes end-to-end pipeline design, performance tuning, and collaboration with data stakeholders.