Databricks Architect: Build Lakehouse & AI Pipelines

Vivanti Consulting; Sydney; On-site; AUD 150,000 - 230,000; Full time

Benefits offered by this job

Mentoring Program

Paid Certifications

Hackathons & Innovation Days

Socials & Team Days

Kudos & Recognition

Job summary

Vivanti Consulting is seeking a Databricks Specialist to architect and implement modern Lakehouse architectures, turning large datasets into high-performance analytics and AI-ready environments.

You will build and optimise enterprise pipelines, migrate legacy estates to Databricks, and collaborate with data scientists to operationalise ML models using MLflow and Databricks Model Serving. Strong Spark/SQL, Python, and cloud fluency are required.

Qualifications

• Hands-on Databricks experience with production pipelines.
• Experience migrating data warehouses to modern Databricks environments.
• Strong SQL and Python (PySpark) skills in large-scale datasets.

Responsibilities

• Architect and optimise enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Develop resilient batch and streaming pipelines using PySpark and Structured Streaming.
• Collaborate with data scientists and ML engineers to operationalise models via MLflow and Databricks Model Serving.
• Tune performance and implement cost-governance across client clusters.
• Establish governance, RBAC, and security in Unity Catalog.

Skills

Databricks

PySpark

SQL

Python

Scala

Cloud platforms

Communication

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

dbt

Kafka

Fivetran

Job description

Vivanti Consulting is seeking a Databricks Specialist to architect and implement modern Lakehouse architectures, turning large datasets into high-performance analytics and AI-ready environments.

You will build and optimise enterprise pipelines, migrate legacy estates to Databricks, and collaborate with data scientists to operationalise ML models using MLflow and Databricks Model Serving. Strong Spark/SQL, Python, and cloud fluency are required.