Databricks Practice Lead: Architect Lakehouse AI Pipelines

Vivanti Consulting; Western Australia; On-site; AUD 180,000 - 260,000; Full time

Benefits offered by this job

Mentoring Program

360° Progress Reviews

Guilds & Training

Job summary

Vivanti is seeking a highly capable Databricks Practice Lead to architect and drive modern Lakehouse architectures across large datasets for analytics and AI.

You will lead the design, development, and optimization of Delta Lake and Unity Catalog platforms, and collaborate with data scientists and analytics teams to operationalise ML models in a fast-paced, client-focused environment.

Qualifications

• Databricks experience and hands-on data pipelines in production.
• Experience with Delta Lake and Unity Catalog is desirable.
• Strong Spark and SQL with Python (PySpark) and Scala as a bonus.

Responsibilities

• Architect, build and optimise enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Develop resilient batch and streaming pipelines using Spark and SDP.
• Migrate legacy data warehouses to Databricks in cloud environments.
• Collaborate with data scientists and ML engineers to operationalise ML models via MLflow.
• Tune workspace/query performance and implement cost governance for large clusters.
• Establish governance and security in Unity Catalog (RBAC, lineage, row/column security).

Skills

Databricks experience

Delivery experience with data

Modern data stack

Spark & SQL

Python (PySpark)

Scala

Cloud fluency

Medallion architecture

Cost & performance instincts

Client-ready communication

Tools

Airflow

dbt

Kafka

Fivetran

Delta Lake

Unity Catalog

Databricks Workflows

Job description

Vivanti is seeking a highly capable Databricks Practice Lead to architect and drive modern Lakehouse architectures across large datasets for analytics and AI.

You will lead the design, development, and optimization of Delta Lake and Unity Catalog platforms, and collaborate with data scientists and analytics teams to operationalise ML models in a fast-paced, client-focused environment.