Data Engineer

Viable Solutions Pty Ltd; City of Melbourne; Hybrid; AUD 150,000 - 230,000; Full time

Job summary

Viable Solutions Pty Ltd in Melbourne, VIC, seeks an experienced Databricks Engineer to design, develop, and optimise data pipelines and lakehouse architectures on the Databricks platform. You will work on large-scale data processing, ML pipelines, and cloud-native data solutions for enterprise clients.

The role requires strong Databricks expertise, Spark, Python, and SQL, with hands-on experience across AWS/Azure/GCP and CI/CD.

Qualifications

• 5–8+ years of experience in data engineering or a related role.
• Strong hands-on experience with Databricks Lakehouse Platform (mandatory).
• Strong proficiency in Apache Spark (PySpark, Spark SQL, Spark Structured Streaming).
• Strong proficiency in Python for data engineering and pipeline development (mandatory).
• Experience with Delta Lake — table management, ACID transactions, time travel.
• Experience with Delta Live Tables (DLT) and Databricks Workflows.
• Strong SQL skills for complex querying and data transformation.
• Experience with Unity Catalog — data governance and access control.
• Experience with MLflow — experiment tracking and model registry.
• Hands-on experience with cloud platforms AWS, Azure, or GCP.
• Experience with CI/CD tools (GitHub Actions, Azure DevOps, Jenkins).
• Experience with Terraform as code for infrastructure.
• Experience working in Agile / Scrum delivery environments.

Responsibilities

• Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform.
• Build and optimise Apache Spark jobs for large-scale data processing and transformation.
• Develop and maintain Delta Lake tables — schema management, optimisation, and time travel.
• Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration.
• Manage and optimise Databricks clusters — configuration, autoscaling, and cost management.
• Develop notebooks and reusable libraries using Python, Scala, or SQL.
• Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale.
• Work with structured, semi-structured, and unstructured data sources.
• Implement lakehouse architecture patterns — Bronze, Silver, Gold layers.
• Integrate Databricks with upstream and downstream systems — databases, APIs, data warehouses.
• Implement data quality checks, validation, and observability across pipelines.
• Manage Unity Catalog for data governance, lineage, and access control.

Skills

Databricks Lakehouse Platform

Apache Spark

Python

SQL

Delta Lake

Delta Live Tables (DLT)

Databricks Workflows

Unity Catalog

MLflow

Cloud platforms (AWS/Azure/GCP)

CI/CD

Terraform

Agile / Scrum

Tools

Databricks Asset Bundles

GitHub Actions

Jenkins

Azure DevOps

Job description

Job Title: Databricks Engineer

Location: Melbourne, VIC (Hybrid)

Experience: 5–8+ Years

About the Role

Viable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.

Key Responsibilities

Databricks Platform Development

• Design, develop, and maintain data pipelines and workflows on the Databricks Lakehouse Platform
• Build and optimise Apache Spark jobs for large-scale data processing and transformation
• Develop and maintain Delta Lake tables — schema management, optimisation, and time travel
• Implement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestration
• Manage and optimise Databricks clusters — configuration, autoscaling, and cost management
• Develop notebooks and reusable libraries using Python, Scala, or SQL

Data Engineering & Pipelines

• Design and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scale
• Work with structured, semi-structured, and unstructured data sources
• Implement lakehouse architecture patterns — Bronze, Silver, and Gold layers
• Integrate Databricks with upstream and downstream systems — databases, APIs, and data warehouses
• Implement data quality checks, validation, and observability across pipelines
• Manage Unity Catalog for data governance, lineage, and access control

Machine Learning & Analytics

• Build and manage MLflow experiments, model tracking, and model registry
• Support data scientists in operationalising ML models on Databricks
• Develop feature engineering pipelines for ML workloads
• Implement Databricks AutoML and experiment management best practices

Cloud & DevOps

• Deploy and manage Databricks workspaces on AWS, Azure, or GCP
• Implement infrastructure as code for Databricks — Terraform or Databricks Asset Bundles
• Build and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or Jenkins
• Implement GitOps practices for notebook and pipeline version control
• Monitor and optimise Databricks workloads for performance and cost efficiency

Governance & Security

• Implement Unity Catalog for data governance, metadata management, and access control
• Ensure data lineage, traceability, and compliance across all data assets
• Apply row-level and column-level security across Delta Lake tables
• Document data models, pipeline architectures, and operational runbooks

Required Skills & Experience

• 5–8+ years of experience in data engineering or a related role
• Strong hands-on experience with Databricks Lakehouse Platform (mandatory)
• Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)
• Strong proficiency in Python — data engineering and pipeline development (mandatory)
• Experience with Delta Lake — table management, optimisation, ACID transactions, and time travel
• Experience with Delta Live Tables (DLT) and Databricks Workflows
• Strong SQL skills — complex querying and data transformation
• Experience with Unity Catalog — data governance and access control
• Experience with MLflow — experiment tracking and model registry
• Hands-on experience with cloud platforms — AWS, Azure, or GCP
• Experience with CI/CD tools — GitHub Actions, Azure DevOps, or Jenkins
• Experience with Terraform for infrastructure as code
• Experience working in Agile / Scrum delivery environments