Senior Data Engineer | Databricks

Vivanti Consulting; Brisbane; In loco; AUD 150.000 - 190.000; Tempo pieno

Vantaggi offerti da questo lavoro

Mentoring Program

Paid Certifications

Hackathons

Regular Training

Annual Progress Reviews

Team Socials

Descrizione del lavoro

Vivanti, a global AI and data consultancy, is seeking a Senior Data Engineer (Databricks) to architect and run Lakehouse platforms, turning large datasets into high-performance analytics and AI engines.

You will work with Delta Lake, Unity Catalog, and Databricks Workflows to migrate, optimize, and govern data across client environments, collaborating with data scientists, ML engineers and analytics teams to operationalise models.

Competenze

• Databricks experience – Data Engineer Professional certification is ideal, but if you’re not certified, we’ll pay for it, and expect demonstrable hands-on Databricks experience.
• Proven delivery experience with data pipelines in production, not just in a lab.
• Hands-on with modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration/transformation tools (Airflow, dbt, Kafka, Fivetran).
• Advanced Spark and SQL with strong Python (PySpark); Scala is a bonus.
• Cloud fluency with Azure, AWS or GCP.
• Medallion Architecture exposure and awareness of gold/silver/bronze layers.
• Cost and performance awareness to ship only when appropriate.
• Strong client-ready communication to explain trade-offs to non-technical stakeholders.

Mansioni

• Architect, build and optimise enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Develop resilient batch and streaming data pipelines using PySpark and Structured Streaming.
• Migrate legacy data warehouses and Hadoop estates to modern Databricks environments.
• Collaborate with data scientists and ML engineers to operationalise ML models with MLflow and Databricks Model Serving.
• Tune workspace and query performance; implement cost governance on large client clusters.
• Establish governance and security in Unity Catalog (RBAC, lineage, row/column security).

Conoscenze

Databricks

PySpark

SQL

Python

Scala

Cloud platforms

Strumenti

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

dbt

Kafka

Descrizione del lavoro

You’ve got the skills. The curiosity. The hunger to grow, try new things and make a real impact.

We’re a growing global AI and data consultancy. Some of the world’s biggest businesses turn to us to help bring their data, AI, cloud and digital solutions to life.

We’re the home base for talented technologists who want more and are looking to take their career to the next level by having more learning, more variety, more support and more chances to do meaningful work.

If that sounds like the kind of place you’d thrive in, keep reading.

The Role…

As a Senior Data Engineer (Databricks), you’re the architect and execution engine, shaping modern Lakehouse architectures and turning massive, complex datasets into high-performance engines for analytics and AI.

What you might work on:

• Architecting, building and optimising enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Developing resilient batch and streaming pipelines using PySpark, Spark Declarative Pipelines (SDP), Structured Streaming and Auto Loader.
• Migrating legacy data warehouses and Hadoop estates onto modern Databricks environments.
• Collaborating closely with data scientists, ML engineers, and analytics teams to operationalise ML models via MLflow and Databricks Model Serving.
• Tuning workspace and query performance, and implementing cost‑governance practices across large-scale client clusters.
• Establishing rock‑solid governance and security in Unity Catalog (RBAC, lineage, row/column‑level security).

What We’re looking for from you:

• Databricks experience – Data Engineer Professional certification is ideal, but if you’re not certified, we’ll pay for it, but we’ll expect demonstrable hands‑on Databricks experience.
• Real delivery experience with data – you’ve built and shipped pipelines in production, not just in a lab.
• Hands‑on with a modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration and/or transformation tooling (Airflow, dbt, Kafka, Fivetran, etc).
• Advanced Spark and SQL – non‑negotiable. Strong Python (PySpark), plus Scala or is a bonus.
• Cloud fluency – working knowledge of at least one of Azure, AWS or GCP.
• Medallion Architecture - you’ve had exposure to the gold, silver, bronze approach and can articulate its benefits.
• Cost and performance instincts – you understand that a job that finishes isn’t the same as a job that should ship.
• Client‑ready communication – you can explain a technical trade‑off to a non‑technical stakeholder in language they understand.

What You’ll Get

Your work matters and so do you. That’s why we back your skills with a structure that supports your development, celebrates your wins, and helps you keep growing - professionally and personally.

Here’s what's waiting for you:

• A platform for growth: Work across industries, clients and technologies that stretch your thinking and build your personal and professional skills.
• Mentoring Program: With a Senior Support Lead
• 360 Progress Reviews: Annual reviews based on honest feedback and clear goals.
• Guilds & Regular Training: Weekly training and learning from your peers and our key vendor partners
• Paid Certifications: Across all major technologies
• Hackathons & Innovation Days: Challenge yourself, build cool stuff
• Kudos & Recognition Programs: We celebrate wins, reward impact and offer long‑term incentives
• Socials & Team Days: Monthly "fun for the sake of it" days, Friday socials, video games, board games, and an F1 racing simulator

Final Word

If you’re looking for the place where your growth will go into overdrive, where you’ll work with great humans, get access to cutting‑edge tools and genuinely enjoy the ride…

Welcome to Vivanti.

The AI Native People Shaping Tomorrow

Vivanti was founded as a haven for talented technologists who want to make a real impact for their clients. Our mission is to transform businesses and to create a smarter, more connected world through the power of data and technology. Together, we are making the future happen - today.