Azure Databricks Data Engineer: Real-Time Pipelines DevOps

CareCone Group; Sydney; On-site; AUD 150,000 - 200,000; Full time

Job summary

CareCone Group in Sydney is seeking a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL, with a focus on scalable, robust data platforms.

You will develop ETL/ELT processes following medallion architecture (Bronze/Silver/Gold) and Delta Lake, deliver real-time and near real-time integrations, and implement CI/CD, secret management, and observability to support analytics across the business.

Qualifications

• Experience designing high-throughput data pipelines on Azure Databricks.
• Experience with medallion architecture (Bronze/Silver/Gold) and Delta Lake.
• Real-time and near real-time data integrations experience.
• Operationalize workloads with scheduling, CI/CD (Git/Azure DevOps), secrets management, and observability.
• Excellent communication and stakeholder management skills.

Responsibilities

• Design, build, and optimize high-throughput data pipelines on Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala) and SQL.
• Implement real-time integrations and ensure observability and reliability.
• Operationalize workloads with scheduling, CI/CD, and secrets management, including governance.
• Collaborate with data stakeholders to ensure data quality and usability.

Skills

Data pipelines

Real-time processing

Azure Databricks

Spark (Scala)

SQL

Python

Tools

Azure Databricks

Spark

Kafka

GitHub

Confluence

Jira

Job description

CareCone Group in Sydney is seeking a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL, with a focus on scalable, robust data platforms.

You will develop ETL/ELT processes following medallion architecture (Bronze/Silver/Gold) and Delta Lake, deliver real-time and near real-time integrations, and implement CI/CD, secret management, and observability to support analytics across the business.