Data Engineer- Azure Databricks

Renaissance Infosystems Pte Ltd; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Renaissance Info Systems is hiring a Data Engineer for Azure Databricks in Sydney. You will design, build, and optimize scalable data pipelines using Spark (Scala) and SQL, and implement ETL/ELT processes on Azure Databricks.

The role requires hands-on development experience with real-time data integration, data warehousing concepts, and CI/CD practices. Permanent, on-site position based in Sydney.

Qualifications

• Proven hands-on experience with Azure Databricks, Spark (Scala) and SQL to design data pipelines.
• Strong Python skills and ability to implement ETL/ELT processes on the Azure platform.
• Experience with real-time/near real-time data integration and CI/CD pipelines.

Skills

Azure Databricks

Scala Spark

SQL

Python

Tools

Databricks

Data warehousing

Azure

GitHub

Job description

About Us

Renaissance Info Systems is a technology and digital recruitment agency, connecting contract and permanent professionals with clients across Asia-Pacific.

We aim to differentiate ourselves through our level of responsiveness, and our understanding that comes from being an IT recruitment agency from the IT Industry. Our recruiters balance sophisticated and simple inter-personal techniques to assure a strong candidate network.

Hiring for Data Engineer- Azure Databricks

Role/Title: Data Engineer- Azure Databricks - 6-12 Years

Mode of Hiring: Permanent

Location: Sydney

Mandatory skills

• Azure Databricks
• Scala Spark
• SQL
• Python

Good to have skills

• Databricks, SQL, Python, Data warehousing concepts
• Detailed Job description (Role & Responsibility, Skill and Experience range)
• We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
• Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
• Proven expertise in designing real time and near real time integrations
• Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.
• Expertise in cloud platforms and related technologies, specifically:
• Hyperscalars Azure, AWS
• Data Platforms '¿ (e.g. Databricks) and big data technologies.
• Big Data Hive, Impala, Sqoop, Spark (Scala/ Python)
• Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
• CICD tools '¿ GitHub, Confluence, Jira, Jenkins
• Proficiency in SQL, Python, and data integration tools.
• Familiarity with Master Data Management solutions
• Excellent communication and stakeholder management skills.

Note: No Visa Sponsorship