Senior Data Engineer (Databricks) — Lakehouse Architect & Builder

Vivanti Consulting; City of Brisbane; On-site; AUD 140,000 - 180,000; Full time

Benefits offered by this job

Growth platform

Mentoring

Progress reviews

Guilds & Training

Paid Certifications

Hackathons

Recognition programs

Socials & Team days

Job summary

Vivanti Consulting seeks a Senior Data Engineer (Databricks) to architect Lakehouse platforms and transform massive datasets into high-performance analytics engines.

You will build resilient batch and streaming pipelines, migrate legacy environments, and collaborate with data scientists and ML engineers to operationalise models using MLflow and Databricks Model Serving. The role emphasizes a modern data stack and strong governance.

Qualifications

• Proven production data pipeline experience.
• Hands-on with a modern data stack: Delta Lake, Unity Catalog, Databricks Workflows.
• Strong Spark and SQL, Python; Scala bonus.
• Cloud fluency across Azure, AWS or GCP.
• Medallion Architecture exposure and understanding.
• Ability to explain technical trade-offs to non-technical stakeholders.

Responsibilities

• Architect and build enterprise Lakehouse platforms on Delta Lake/Unity Catalog.
• Develop resilient batch and streaming pipelines using PySpark and Structured Streaming.
• Migrate legacy data warehouses and Hadoop estates to Databricks environments.
• Collaborate with data scientists and ML engineers to operationalise models (MLflow).
• Tune performance and implement cost-governance across large-scale clusters.

Skills

Databricks

Spark

Python

Scala

Cloud

Delta Lake

Unity Catalog

Airflow

dbt

Kafka

MLflow

Model Serving

Tools

Databricks Workflows

Delta Lake (storage)

Unity Catalog

Airflow

dbt

Kafka

Job description

Vivanti Consulting seeks a Senior Data Engineer (Databricks) to architect Lakehouse platforms and transform massive datasets into high-performance analytics engines.

You will build resilient batch and streaming pipelines, migrate legacy environments, and collaborate with data scientists and ML engineers to operationalise models using MLflow and Databricks Model Serving. The role emphasizes a modern data stack and strong governance.