Azure Data Engineer

CareCone Group; Sydney; On-site; AUD 120,000 - 160,000; Full time

Job summary

CareCone Group is seeking a hands-on Developer Programmer to design and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. The role focuses on scalable ETL/ELT development following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.

The ideal candidate will implement real-time and near real-time integrations, operationalize workloads with scheduling, CI/CD using Git, secrets management, and strong observability across cloud data

Qualifications

• Hands-on developer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL.
• Design & develop scalable ETL/ELT pipelines following medallion architecture (Bronze/Silver/Gold) and Delta Lake.
• Proven expertise in real-time and near real-time integrations.
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git, secrets management, and observability.
• Expertise in cloud platforms and data platforms like Databricks.

Responsibilities

• Design, build and optimize high-throughput data pipelines on Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala) and SQL.
• Implement real-time and near real-time data integrations.
• Operationalize workloads with scheduling, CI/CD, and observability.

Skills

Azure Databricks

Spark (Scala)

SQL

Python

Kafka

Delta Lake

CI/CD (Git/Azure DevOps)

Observability

Tools

Git

Azure DevOps

Databricks

Spark Streaming

Delta Lake

Job description

Detailed Job description (Role & Responsibility, Skill and Experience range)

• We are looking for a hands-on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.
• Expertise in cloud platforms and related technologies, specifically:
• Data Platforms (e. g. Databricks) and big data technologies.
• Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
• Proficiency in SQL, Python, and data integration tools.
• Familiarity with Master Data Management solutions
• Excellent communication and stakeholder management skills.