Data Engineer

CareCone Group; Sydney; On-site; AUD 120,000 - 160,000; Full time

Job summary

CareCone Group in Sydney is seeking a hands-on Developer to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. The role focuses on ETL/ELT architectures, Delta Lake best practices, and real-time integrations.

You will implement CI/CD, secrets management, and observability while working with modern data platforms. Key skills include Scala Spark, SQL, and Python, plus familiarity with Databricks and Kafka.

Qualifications

• Hands-on developer to design and optimize high-throughput data pipelines on Azure Databricks with Spark (Scala) and SQL.
• Design ETL/ELT pipelines following Delta Lake best practices.
• Real-time and near real-time data integrations.
• CI/CD with Git (Azure DevOps), secrets management, and observability.
• Familiarity with Databricks, Spark Streaming and Kafka.
• Strong SQL and Python skills, plus stakeholder communication.

Responsibilities

• Design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL.
• Develop scalable ETL/ELT pipelines with medallion architecture and Delta Lake.
• Implement real-time and near real-time data integrations.
• Operationalize workloads with scheduling, CI/CD, secrets management, and observability.

Skills

Scala

Spark

SQL

Python

Data warehousing concepts

Tools

Databricks

Job description

Location : Sydney

Mode:Permanent

Job Description:

Primary or Mandatory skills

2. Scala Spark

3. SQL

4. Python

Good to have skills

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

• We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.

Expertise in cloud platforms and related technologies, specifically:

Data Platforms (e.g. Databricks) and big data technologies.

Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka

• Proficiency in SQL, Python, and data integration tools.
• Familiarity with Master Data Management solutions
• Excellent communication and stakeholder management skills.