Data Engineer

Viable Solutions Pty Ltd; City of Melbourne; On-site; AUD 120,000 - 180,000; Full time

Job summary

Viable Solutions Pty Ltd is seeking an experienced Databricks Engineer to design, develop and optimise data pipelines and lakehouse architectures on the Databricks platform for enterprise clients.

You will build Spark jobs, manage Delta Lake tables, implement DLT and workflows, and collaborate with data scientists to operationalise ML models in cloud environments (AWS/Azure/GCP).

Qualifications

• 5–8+ years of experience in data engineering or a related role.
• Mandatory: Databricks Lakehouse Platform.
• Mandatory: Apache Spark (PySpark, Spark SQL, Structured Streaming).
• Mandatory: Python for data engineering and pipeline development.
• Experience with Delta Lake, Delta Live Tables (DLT), and Databricks Workflows.
• Strong SQL skills for complex data transformations.
• Hands-on cloud experience (AWS, Azure, or GCP).
• CI/CD tooling experience (GitHub Actions, Azure DevOps, Jenkins).
• Terraform for infrastructure as code.
• Experience in Agile / Scrum delivery environments.

Responsibilities

• Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform.
• Build and optimise Apache Spark jobs for large-scale data processing and transformation.
• Develop and maintain Delta Lake tables with schema management and time travel.
• Implement Databricks Workflows and Delta Live Tables for orchestration.
• Manage Databricks clusters with cost-aware performance tuning.
• Develop notebooks and reusable libraries using Python, Scala, or SQL.
• Design and implement ELT/ETL pipelines for scalable data ingestion.
• Work with structured, semi-structured, and unstructured data sources.
• Implement lakehouse patterns across Bronze/Silver/Gold layers.
• Collaborate with data scientists to operationalise ML models on Databricks.

Skills

Databricks Lakehouse Platform

Apache Spark

Python

Delta Lake

SQL

Unity Catalog

MLflow

CI/CD tools

Terraform

Agile / Scrum

AWS

Azure

GCP

GitHub Actions

Azure DevOps

Jenkins

Delta Live Tables (DLT)

Tools

GitHub Actions

Azure DevOps

Jenkins

Terraform

Delta Live Tables

Job description

Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.

Key Responsibilities

Databricks Platform Development

• Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
• Build and optimise Apache Spark jobs for large-scale data processing and transformation
• Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
• Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
• Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
• Develop notebooks and reusable libraries using Python, Scala, or SQL
• Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
• Work with structured, semi-structured, and unstructured data sources
• Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
• Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
• Implement data quality checks, validation, and observability across pipelines
• Manage Unity Catalog for data governance, lineage, and access control
• Build and manage MLflow experiments, model tracking, and model registry
• Support data scientists in operationalising ML models on Databricks
• Develop feature engineering pipelines for ML workloads
• Implement Databricks AutoML and experiment management best practices
• Deploy and manage Databricks workspaces on AWS, Azure, or GCP
• Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
• Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
• Implement GitOps practices for notebook and pipeline version control
• Monitor and optimise Databricks workloads for performance and cost efficiency

Governance & Security

• Implement Unity Catalog for data governance, metadata management, and access control
• Ensure data lineage, traceability, and compliance across all data assets
• Apply row-level and column-level security across Delta Lake tables
• Document data models, pipeline architectures, and operational runbooks

Required Skills & Experience

• 5–8+ years of experience in data engineering or a related role
• Strong hands‑on experience with Databricks Lakehouse Platform (mandatory)
• Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
• Strong proficiency in Python — data engineering and pipeline development (mandatory)
• Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
• Experience with Delta Live Tables (DLT) and Databricks Workflows
• Strong SQL skills — complex querying and data transformation
• Experience with Unity Catalog — data governance and access control
• Experience with MLflow — experiment tracking and model registry
• Hands‑on experience with cloud platforms — AWS, Azure, or GCP
• Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
• Experience with Terraform for infrastructure as code
• Experience working in Agile / Scrum delivery environments