Databricks Architect: Lead Scalable Data Platform

Unison Group; Sydney; On-site; AUD 140,000 - 210,000; Full time

Job summary

Unison Group is seeking a Databricks Architect to lead the design and implementation of scalable, high-performance data platforms using Databricks and Apache Spark. The role collaborates with data engineers, scientists and business teams to deliver data lakes, lakehouse architectures and analytics solutions enabling business insights.

The candidate will architect and implement Databricks-based data platforms, design end-to-end ingestion and analytics pipelines, and establish best practices for

Qualifications

• Hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Proficient in Python, Scala and SQL.
• Experience designing data lakes, lakehouse architectures and cloud-based ETL/ELT pipelines.
• Strong understanding of data architecture, modeling and governance.
• Excellent analytical, communication and solution design skills.
• Experience working in agile, collaborative environments.

Responsibilities

• Architect and implement Databricks-based data platforms and pipelines.
• Design end-to-end data ingestions, processing and analytics using Databricks and Spark.
• Develop best practices for data engineering, governance and security within Databricks.
• Optimize compute and storage to balance cost and performance.
• Collaborate with cross-functional teams to translate requirements into scalable solutions.

Skills

Databricks

Apache Spark

Delta Lake

Python

Scala

SQL

Data Lakes

Lakehouse

ETL/ELT

AWS

Azure

GCP

Data governance

Agile

Communication

Solution design

Job description

Unison Group is looking for a Databricks Architect to lead the design and implementation of scalable, high-performance data platforms leveraging Databricks and Apache Spark. The successful candidate will collaborate with data engineers, scientists, and business teams to build data lakes, lakehouse architectures, and analytics solutions that drive business insights.

• Architect and implement Databricks-based data platforms and pipelines.
• Design end-to-end solutions for data ingestion, processing, and analytics using Databricks and Spark.
• Develop best practices for data engineering, governance, and security within the Databricks environment.
• Optimize compute and storage resources to balance cost and performance.
• Collaborate closely with cross-functional teams to understand requirements and translate them into scalable solutions.

• Strong hands-on experience with Databricks, Apache Spark, and Delta Lake technologies.
• Proficiency in big data programming languages such as Python, Scala, and SQL.
• Experience designing and deploying data lakes, lakehouse architectures, and ETL/ELT pipelines on cloud platforms (AWS, Azure, or GCP).
• Deep understanding of data architecture, data modeling, and data governance.
• Excellent analytical, communication, and solution design skills.
• Prior experience working in agile, collaborative environments.