Data Engineer (Azure Databricks)

CareCone Group; Sydney; On-site; AUD 150,000 - 200,000; Full time

Job summary

CareCone Group in Sydney is seeking a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL, with a focus on scalable, robust data platforms.

You will develop ETL/ELT processes following medallion architecture (Bronze/Silver/Gold) and Delta Lake, deliver real-time and near real-time integrations, and implement CI/CD, secret management, and observability to support analytics across the business.

Qualifications

• Experience designing high-throughput data pipelines on Azure Databricks.
• Experience with medallion architecture (Bronze/Silver/Gold) and Delta Lake.
• Real-time and near real-time data integrations experience.
• Operationalize workloads with scheduling, CI/CD (Git/Azure DevOps), secrets management, and observability.
• Excellent communication and stakeholder management skills.

Responsibilities

• Design, build, and optimize high-throughput data pipelines on Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala) and SQL.
• Implement real-time integrations and ensure observability and reliability.
• Operationalize workloads with scheduling, CI/CD, and secrets management, including governance.
• Collaborate with data stakeholders to ensure data quality and usability.

Skills

Data pipelines

Real-time processing

Azure Databricks

Spark (Scala)

SQL

Python

Tools

Azure Databricks

Spark

Kafka

GitHub

Confluence

Jira

Job description

• We are looking for a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.

Expertise in cloud platforms and related technologies, specifically:

• Data Platforms (e.g. Databricks) and big data technologies.
• Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
• CICD tools GitHub, Confluence, Jira, Jenkins
• Proficiency in SQL, Python, and data integration tools.
• Familiarity with Master Data Management solutions
• Excellent communication and stakeholder management skills.