Data Engineer- Azure Databricks

Renaissance Infosystems Pte Ltd; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Renaissance Infosystems Pte Ltd in Sydney is seeking a Data Engineer with 6–12 years of experience to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark (Scala) and SQL. This permanent role is based in Sydney and requires hands-on expertise in data engineering.

You will implement medallion architecture with Delta Lake, develop real-time and near real-time integrations, and oversee CI/CD practices with GitHub/Azure DevOps, ensuring robust data workflows and

Qualifications

• Hands-on developer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL.
• Design and develop scalable ETL/ELT pipelines in Azure Databricks following medallion (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in real-time and near real-time integrations and CI/CD with GitHub, Azure DevOps, and observability.

Skills

Azure Databricks

Scala Spark

SQL

Python

Tools

GitHub

Confluence

Jira

Jenkins

Azure DevOps

Delta Lake

Job description

About Us

Renaissance Info Systems is a technology and digital recruitment agency, connecting contract and permanent professionals with clients across Asia-Pacific.

We aim to differentiate ourselves through our level of responsiveness, and our understanding that comes from being an IT recruitment agency from the IT Industry. Our recruiters balance sophisticated and simple inter-personal techniques to assure a strong candidate network.

Know More: http://www.reninfo.com.au

Role

Data Engineer- Azure Databricks - 6-12 Years

Mode of Hiring

Permanent

Location

Sydney

Mandatory skills

• Azure Databricks
• Scala Spark
• SQL
• Python

Good to have skills

• Databricks, SQL, Python, Data warehousing concepts
• Detailed Job description (Role & Responsibility, Skill and Experience range)
• We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.
• Expertise in cloud platforms and related technologies, specifically:
• Hyperscalars Azure, AWS
• Data Platforms '¿ (e.g. Databricks) and big data technologies.
• Big Data Hive, Impala, Sqoop, Spark (Scala/ Python)
• Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
• CICD tools '¿ GitHub, Confluence, Jira, Jenkins
• Proficiency in SQL, Python, and data integration tools.
• Familiarity with Master Data Management solutions
• Excellent communication and stakeholder management skills.

Note: No Visa Sponsorship

No Visa Sponsorship

Thanks and Regards,

Nidhi Anand

Recruitment Consultant

Renaissance InfoSystem

C: 0423430***

E: ***email_hidden***

W: http://www.reninfo.com.au