Databricks Data Engineer — Pipelines & Governance (Hybrid)

National Heavy Vehicle Regulator; City of Brisbane; Hybrid; AUD 115,000 - 156,000; Full time

Benefits offered by this job

Hybrid work

Wellbeing programs

Salary packaging

Professional development

Job summary

National Heavy Vehicle Regulator in Brisbane is seeking a Data Engineer to design and operate scalable data pipelines in Databricks, with strong Python and Spark SQL skills. You will help safeguard data with Unity Catalog and work within an Agile team to deliver on the Data Science and Analytics strategy.

The role emphasizes governance, metadata-driven frameworks, and cost optimisation, with hybrid work options and a competitive salary.

Qualifications

• Minimum four years in data engineering with cloud platforms.
• Experience with Databricks or equivalent is essential.
• Bachelor's degree in a related field preferred.

Responsibilities

• Design, build and maintain scalable data pipelines using Databricks.
• Implement and manage Unity Catalog security and governance.
• Collaborate in a Scrum team to meet roadmap milestones.
• Deliver CI/CD ready code with testing and reviews.

Skills

Databricks

Python

PySpark

Spark SQL

Delta Lake

Unity Catalog

PII handling

GIT/CI-CD

Azure data services

Data modelling

Agile Scrum

Education

Bachelor's degree

Tools

Lakeflow

Auto Loader

MLflow

Delta Live Tables

Job description

National Heavy Vehicle Regulator in Brisbane is seeking a Data Engineer to design and operate scalable data pipelines in Databricks, with strong Python and Spark SQL skills. You will help safeguard data with Unity Catalog and work within an Agile team to deliver on the Data Science and Analytics strategy.

The role emphasizes governance, metadata-driven frameworks, and cost optimisation, with hybrid work options and a competitive salary.