Databricks Architect

Unison Group; Sydney; On-site; AUD 150,000 - 190,000; Full time

Job summary

Unison Group is seeking a Databricks Architect in Sydney to lead design and implementation of scalable data platforms leveraging Databricks and Apache Spark. You will collaborate with data engineers, scientists, and business teams to build data lakes, lakehouse architectures, and analytics solutions that drive business insights.

Responsibilities include architecting pipelines, end-to-end data solutions, governance and security practices, and cost-performance optimization in cloud environments

Qualifications

• Strong hands-on experience with Databricks, Apache Spark, and Delta Lake technologies.
• Proficiency in Python, Scala, and SQL.
• Experience designing and deploying data lakes, lakehouse architectures, and ETL/ELT pipelines on cloud platforms.
• Deep understanding of data architecture, modeling, and governance.
• Excellent analytical and communication skills.
• Experience working in agile, collaborative environments.

Responsibilities

• Architect and implement Databricks-based data platforms and pipelines.
• Design end-to-end data ingestion, processing, and analytics solutions.
• Develop best practices for data governance and security in Databricks.
• Optimize compute and storage to balance cost and performance.
• Collaborate with cross-functional teams to translate requirements into scalable solutions.

Skills

Databricks

Apache Spark

Delta Lake

Python

Scala

SQL

Data Lakes

Lakehouse Architecture

Job description

Unison Group is looking for a Databricks Architect to lead the design and implementation of scalable, high-performance data platforms leveraging Databricks and Apache Spark. The successful candidate will collaborate with data engineers, scientists, and business teams to build data lakes, lakehouse architectures, and analytics solutions that drive business insights.

• Architect and implement Databricks-based data platforms and pipelines.
• Design end-to-end solutions for data ingestion, processing, and analytics using Databricks and Spark.
• Develop best practices for data engineering, governance, and security within the Databricks environment.
• Optimize compute and storage resources to balance cost and performance.
• Collaborate closely with cross-functional teams to understand requirements and translate them into scalable solutions.

Requirements

• Strong hands-on experience with Databricks, Apache Spark, and Delta Lake technologies.
• Proficiency in big data programming languages such as Python, Scala, and SQL.
• Experience designing and deploying data lakes, lakehouse architectures, and ETL/ELT pipelines on cloud platforms (AWS, Azure, or GCP).
• Deep understanding of data architecture, data modeling, and data governance.
• Excellent analytical, communication, and solution design skills.
• Prior experience working in agile, collaborative environments.