Senior Databricks Data Engineer - Lakehouse Architect

Vivanti Consulting; Brisbane; On-site; AUD 150,000 - 190,000; Full time

Benefits offered by this job

Mentoring Program

Paid Certifications

Hackathons

Regular Training

Annual Progress Reviews

Team Socials

Job summary

Vivanti, a global AI and data consultancy, is seeking a Senior Data Engineer (Databricks) to architect and run Lakehouse platforms, turning large datasets into high-performance analytics and AI engines.

You will work with Delta Lake, Unity Catalog, and Databricks Workflows to migrate, optimize, and govern data across client environments, collaborating with data scientists, ML engineers and analytics teams to operationalise models.

Qualifications

• Databricks experience – Data Engineer Professional certification is ideal, but if you’re not certified, we’ll pay for it, and expect demonstrable hands-on Databricks experience.
• Proven delivery experience with data pipelines in production, not just in a lab.
• Hands-on with modern data stack – Delta Lake, Unity Catalog, Databricks Workflows, and orchestration/transformation tools (Airflow, dbt, Kafka, Fivetran).
• Advanced Spark and SQL with strong Python (PySpark); Scala is a bonus.
• Cloud fluency with Azure, AWS or GCP.
• Medallion Architecture exposure and awareness of gold/silver/bronze layers.
• Cost and performance awareness to ship only when appropriate.
• Strong client-ready communication to explain trade-offs to non-technical stakeholders.

Responsibilities

• Architect, build and optimise enterprise Lakehouse platforms on Delta Lake and Unity Catalog.
• Develop resilient batch and streaming data pipelines using PySpark and Structured Streaming.
• Migrate legacy data warehouses and Hadoop estates to modern Databricks environments.
• Collaborate with data scientists and ML engineers to operationalise ML models with MLflow and Databricks Model Serving.
• Tune workspace and query performance; implement cost governance on large client clusters.
• Establish governance and security in Unity Catalog (RBAC, lineage, row/column security).

Skills

Databricks

PySpark

SQL

Python

Scala

Cloud platforms

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

dbt

Kafka

Job description

Vivanti, a global AI and data consultancy, is seeking a Senior Data Engineer (Databricks) to architect and run Lakehouse platforms, turning large datasets into high-performance analytics and AI engines.

You will work with Delta Lake, Unity Catalog, and Databricks Workflows to migrate, optimize, and govern data across client environments, collaborating with data scientists, ML engineers and analytics teams to operationalise models.