Databricks Specialist

Vivanti Consulting; Sydney; On-site; AUD 150,000 - 230,000; Full time

Benefits offered by this job

Mentoring Program

Paid Certifications

Hackathons & Innovation Days

Socials & Team Days

Kudos & Recognition

Job summary

Vivanti Consulting is seeking a Databricks Specialist to architect and implement modern Lakehouse architectures, turning large datasets into high-performance analytics and AI-ready environments.

You will build and optimise enterprise pipelines, migrate legacy estates to Databricks, and collaborate with data scientists to operationalise ML models using MLflow and Databricks Model Serving. Strong Spark/SQL, Python, and cloud fluency are required.

Qualifications

• Hands-on Databricks experience with production pipelines.
• Experience migrating data warehouses to modern Databricks environments.
• Strong SQL and Python (PySpark) skills in large-scale datasets.

Responsibilities

• Architect and optimise enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Develop resilient batch and streaming pipelines using PySpark and Structured Streaming.
• Collaborate with data scientists and ML engineers to operationalise models via MLflow and Databricks Model Serving.
• Tune performance and implement cost-governance across client clusters.
• Establish governance, RBAC, and security in Unity Catalog.

Skills

Databricks

PySpark

SQL

Python

Scala

Cloud platforms

Communication

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

dbt

Kafka

Fivetran

Job description

You’ve got the skills. The curiosity. The hunger to grow, try new things and make a real impact.

We’re a growing global AI and data consultancy. Some of the world’s biggest businesses turn to us to help bring their data, AI, cloud and digital solutions to life.

We’re the home base for talented technologists who want more and are looking to take their career to the next level by having more learning, more variety, more support and more chances to do meaningful work.

If that sounds like the kind of place you’d thrive in, keep reading.

The Role...

As a Databricks Specialist, you’re the architect and execution engine, shaping modern Lakehouse architectures and turning massive, complex datasets into high-performance engines for analytics and AI.

What you might work on:

• Architecting, building and optimising enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Developing resilient batch and streaming pipelines using PySpark, Spark Declarative Pipelines (SDP), Structured Streaming and Auto Loader.
• Migrating legacy data warehouses and Hadoop estates onto modern Databricks environments.
• Collaborating closely with data scientists, ML engineers, and analytics teams to operationalise ML models via MLflow and Databricks Model Serving.
• Tuning workspace and query performance, and implementing cost-governance practices across large-scale client clusters.
• Establishing rock-solid governance and security in Unity Catalog (RBAC, lineage, row/column-level security).

What We’re looking for from you:

• Databricks experience – Data Engineer Professional certification is ideal, but if you’re not certified, we’ll pay for it, but we’ll expect demonstrable hands‑on Databricks experience.
• Real delivery experience with data – you’ve built and shipped pipelines in production, not just in a lab.
• Hands‑on with a modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration and/or transformation tooling (Airflow, dbt, Kafka, Fivetran, etc).
• Advanced Spark and SQL – non‑negotiable. Strong Python (PySpark), plus Scala or is a bonus.
• Cloud fluency – working knowledge of at least one of Azure, AWS or GCP.
• Medallion Architecture - you’ve had exposure to the gold, silver, bronze approach and can articulate its benefits.
• Cost and performance instincts – you understand that a job that finishes isn't the same as a job that should ship.
• Client-ready communication – you can explain a technical trade‑off to a non‑technical stakeholder in language they understand.

What You’ll Get

Your work matters and so do you. That's why we back your skills with a structure that supports your development, celebrates your wins, and helps you keep growing – professionally and personally.

Here’s what’s waiting for you:

• A platform for growth: Work across industries, clients and technologies that stretch your thinking and build your personal and professional skills.
• Mentoring Program: With a Senior Support Lead
• 360 Progress Reviews: Annual reviews based on honest feedback and clear goals.
• Guilds & Regular Training: Weekly training and learning from your peers and our key vendor partners
• Paid Certifications: Across all major technologies
• Hackathons & Innovation Days: Challenge yourself, build cool stuff
• Kudos & Recognition Programs: We celebrate wins, reward impact and offer long‑term incentives
• Socials & Team Days: Monthly “fun for the sake of it” days, Friday socials, video games, board games, and soon an F1 racing simulator

Final Word

If you’re looking for the place where your growth will go into overdrive, where you’ll work with great humans, get access to cutting‑edge tools and genuinely enjoy the ride...

Welcome to Vivanti.

The AI Native People Shaping Tomorrow

Vivanti was founded as a haven for talented technologists who want to make a real impact for their clients. Our mission is to transform businesses and to create a smarter, more connected world through the power of data and technology.

Together, we are making the future happen - today.

*We also offer sponsorship opportunities, so whether you’re already based in Australia or considering a move from overseas, we’d still love to hear from you!