Databricks Practice Lead: Architect Lakehouse & AI Pipelines

Vivanti Consulting; Western Australia; In loco; AUD 180.000 - 240.000; Tempo pieno

Vantaggi offerti da questo lavoro

Growth platform

Mentoring programme

360 reviews

Guilds & training

Paid certifications

Hackathons

Kudos & recognition

Socials & team days

Descrizione del lavoro

Vivanti is seeking a Databricks Practice Lead to architect Lakehouse platforms and deliver high-performance analytics solutions. You will build scalable pipelines, migrate legacy warehouses, and collaborate with data scientists to operationalise ML models.

Ideal candidates have hands-on Databricks experience, strong Spark/SQL, and proficiency across Delta Lake and Unity Catalog. Expect governance, cost control, and cross-team collaboration in a dynamic, growth-focused environment.

Competenze

• Demonstrable Databricks experience in production
• Hands-on with modern data stack across platforms
• Experience migrating legacy data warehouses to Databricks
• Strong Spark and SQL skills with Python (PySpark)
• Cloud fluency across Azure, AWS, or GCP

Mansioni

• Architect and deliver Lakehouse architectures at scale
• Build and optimize batch/streaming pipelines
• Migrate legacy data estates to Databricks environments
• Collaborate with data scientists and ML engineers to operationalise models
• Ensure governance, RBAC, and cost governance for large client clusters

Conoscenze

Databricks experience

Delivery experience

Spark & SQL

Python (PySpark)

Cloud knowledge

Medallion architecture

Cost & performance mindset

Client communication

Strumenti

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

dbt

Kafka

Fivetran

Descrizione del lavoro

You’ve got the skills. The curiosity. The hunger to grow, try new things and make a real impact.

We’re a growing global AI and data consultancy. Some of the world’s biggest businesses turn to us to help bring their data, AI, cloud and digital solutions to life.

We’re the home base for talented technologists who want more and are looking to take their career to the next level by having more learning, more variety, more support and more chances to do meaningful work.

If that sounds like the kind of place you’d thrive in, keep reading.

The Role…

As the Databricks Practice Lead, you’re the architect and execution engine, shaping modern Lakehouse architectures and turning massive, complex datasets into high-performance engines for analytics and AI.

What you might work on:

• Architecting, building and optimising enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Developing resilient batch and streaming pipelines using PySpark, Spark Declarative Pipelines (SDP), Structured Streaming and Auto Loader.
• Migrating legacy data warehouses and Hadoop estates onto modern Databricks environments.
• Collaborating closely with data scientists, ML engineers, and analytics teams to operationalise ML models via MLflow and Databricks Model Serving.
• Tuning workspace and query performance, and implementing cost-governance practices across large-scale client clusters.
• Establishing rock-solid governance and security in Unity Catalog (RBAC, lineage, row/column-level security).

What We’re looking for from you:

• Databricks experience – Data Engineer Professional certification is ideal, but if you're not certified, we'll pay for it, but we’ll expect demonstrable hands-on Databricks experience.
• Real delivery experience with data – you've built and shipped pipelines in production, not just in a lab.
• Hands-on with a modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration and/or transformation tooling (Airflow, dbt, Kafka, Fivetran, etc).
• Advanced Spark and SQL – non-negotiable. Strong Python (PySpark), plus Scala or is a bonus.
• Cloud fluency – working knowledge of at least one of Azure, AWS or GCP.
• Medallion Architecture - you’ve had exposure to the gold, silver, bronze approach and can articulate its benefits.
• Cost and performance instincts – you understand that a job that finishes isn't the same as a job that should ship.
• Client-ready communication – you can explain a technical trade-off to a non-technical stakeholder in language they understand.

What You’ll Get

Your work matters and so do you. That's why we back your skills with a structure that supports your development, celebrates your wins, and helps you keep growing – professionally and personally.

Here's what's waiting for you:

• A platform for growth: Work across industries, clients and technologies that stretch your thinking and build your personal and professional skills.
• Mentoring Program: With a Senior Support Lead
• 360° Progress Reviews: Annual reviews based on honest feedback and clear goals.
• Guilds & Regular Training: Weekly training and learning from your peers and our key vendor partners
• Paid Certifications: Across all major technologies
• Hackathons & Innovation Days: Challenge yourself, build cool stuff
• Kudos & Recognition Programs: We celebrate wins, reward impact and offer long-term incentives
• Socials & Team Days: Monthly "fun for the sake of it" days, Friday socials, video games, board games, and an F1 racing simulator

Final Word

If you're looking for the place where your growth will go into overdrive, where you'll work with great humans, get access to cutting-edge tools and genuinely enjoy the ride…

Welcome to Vivanti.

The AI Native People Shaping Tomorrow

Vivanti was founded as a haven for talented technologists who want to make a real impact for their clients. Our mission is to transform businesses and to create a smarter, more connected world through the power of data and technology. Together, we are making the future happen - today.