Senior Data Engineer | Databricks

Vivanti Consulting; City of Brisbane; On-site; AUD 140,000 - 180,000; Full time

Benefits offered by this job

Growth platform

Mentoring

Progress reviews

Guilds & Training

Paid Certifications

Hackathons

Recognition programs

Socials & Team days

Job summary

Vivanti Consulting seeks a Senior Data Engineer (Databricks) to architect Lakehouse platforms and transform massive datasets into high-performance analytics engines.

You will build resilient batch and streaming pipelines, migrate legacy environments, and collaborate with data scientists and ML engineers to operationalise models using MLflow and Databricks Model Serving. The role emphasizes a modern data stack and strong governance.

Qualifications

• Proven production data pipeline experience.
• Hands-on with a modern data stack: Delta Lake, Unity Catalog, Databricks Workflows.
• Strong Spark and SQL, Python; Scala bonus.
• Cloud fluency across Azure, AWS or GCP.
• Medallion Architecture exposure and understanding.
• Ability to explain technical trade-offs to non-technical stakeholders.

Responsibilities

• Architect and build enterprise Lakehouse platforms on Delta Lake/Unity Catalog.
• Develop resilient batch and streaming pipelines using PySpark and Structured Streaming.
• Migrate legacy data warehouses and Hadoop estates to Databricks environments.
• Collaborate with data scientists and ML engineers to operationalise models (MLflow).
• Tune performance and implement cost-governance across large-scale clusters.

Skills

Databricks

Spark

Python

Scala

Cloud

Delta Lake

Unity Catalog

Airflow

dbt

Kafka

MLflow

Model Serving

Tools

Databricks Workflows

Delta Lake (storage)

Unity Catalog

Airflow

dbt

Kafka

Job description

You’ve got the skills. The curiosity. The hunger to grow, try new things and make a real impact.

We’re a growing global AI and data consultancy. Some of the world’s biggest businesses turn to us to help bring their data, AI, cloud and digital solutions to life.

We’re the home base for talented technologists who want more and are looking to take their career to the next level by having more learning, more variety, more support and more chances to do meaningful work.

If that sounds like the kind of place you’d thrive in, keep reading.

The Role…

As a Senior Data Engineer (Databricks), you’re the architect and execution engine, shaping modern Lakehouse architectures and turning massive, complex datasets into high-performance engines for analytics and AI.

What you might work on:

• Architecting, building and optimising enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Developing resilient batch and streaming pipelines using PySpark, Spark Declarative Pipelines (SDP), Structured Streaming and Auto Loader.
• Migrating legacy data warehouses and Hadoop estates onto modern Databricks environments.
• Collaborating closely with data scientists, ML engineers, and analytics teams to operationalise ML models via MLflow and Databricks Model Serving.
• Tuning workspace and query performance, and implementing cost-governance practices across large-scale client clusters.
• Establishing rock-solid governance and security in Unity Catalog (RBAC, lineage, row/column-level security).

What We’re looking for from you:

• Databricks experience – Data Engineer Professional certification is ideal, but if you're not certified, we'll pay for it, but we’ll expect demonstrable hands‑on Databricks experience.
• Real delivery experience with data – you've built and shipped pipelines in production, not just in a lab.
• Hands‑on with a modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration and/or transformation tooling (Airflow, dbt, Kafka, Fivetran, etc).
• Advanced Spark and SQL – non‑negotiable. Strong Python (PySpark), plus Scala or is a bonus.
• Cloud fluency – working knowledge of at least one of Azure, AWS or GCP.
• Medallion Architecture - you’ve had exposure to the gold, silver, bronze approach and can articulate its benefits.
• Cost and performance instincts – you understand that a job that finishes isn't the same as a job that should ship.
• Client‑ready communication – you can explain a technical trade‑off to a non‑technical stakeholder in language they understand.

What You’ll Get

Your work matters and so do you. That's why we back your skills with a structure that supports your development, celebrates your wins, and helps you keep growing – professionally and personally.

Here’s what’s waiting for you:

• A platform for growth: Work across industries, clients and technologies that stretch your thinking and build your personal and professional skills.
• Mentoring Program: With a Senior Support Lead
• 360° Progress Reviews: Annual reviews based on honest feedback and clear goals.
• Guilds & Regular Training: Weekly training and learning from your peers and our key vendor partners
• Paid Certifications: Across all major technologies
• Hackathons & Innovation Days: Challenge yourself, build cool stuff
• Kudos & Recognition Programs: We celebrate wins, reward impact and offer long‑term incentives
• Socials & Team Days: Monthly "fun for the sake of it" days, Friday socials, video games, board games, and an F1 racing simulator

Final Word

If you're looking for the place where your growth will go into overdrive, where you'll work with great humans, get access to cutting‑edge tools and genuinely enjoy the ride…

Welcome to Vivanti.

The AI Native People Shaping Tomorrow

Vivanti was founded as a haven for talented technologists who want to make a real impact for their clients. Our mission is to transform businesses and to create a smarter, more connected world through the power of data and technology. Together, we are making the future happen - today.