Data Engineer

NHVR External Careers; City of Brisbane; Hybrid; AUD 120,000 - 150,000; Full time

Benefits offered by this job

Hybrid working

Salary package + superannuation

Wellbeing programs

Training and development

Discount IT products

Job summary

NHVR is seeking a Data Engineer to join its Data & Technology team in Brisbane. You will design, build, and maintain scalable data pipelines using Databricks, Python, and Spark SQL, and contribute to governance and analytics initiatives across the organisation.

You will work in a Scrum environment, collaborate on BI/analytics, and support data security with Unity Catalog while improving performance and cost management. Hybrid work up to 2 days from home.

Qualifications

• Four+ years of data engineering experience with Databricks or similar cloud platform.
• Bachelor's degree in a relevant field desirable.
• Experience with Agile Scrum, SDLC, and data lifecycle processes.
• Hands-on Databricks experience with notebooks and Jobs/Workflows.
• Strong Python and PySpark pipelines with advanced SQL.
• Experience with Delta Lake features such as MERGE and VACUUM.
• Experience with Unity Catalog security and governance.
• Knowledge of metadata-driven or configuration-driven pipelines.
• Data modeling knowledge (relational, dimensional, medallion).

Responsibilities

• Collaborate in a Scrum team to deliver BAU work and roadmap milestones.
• Design, build and maintain scalable data pipelines in Databricks.
• Optimize performance and cost through cluster configuration and tuning.
• Implement and manage Unity Catalog security and data masking.
• Support incident management, monitoring, and troubleshooting.
• Proactively document risks and impediments for stakeholders.

Skills

Databricks

Python

Spark SQL

SQL

Agile/Scrum

CI/CD

Data governance

Data modeling

Security/compliance

Education

Bachelor's degree in a relevant field

Tools

Delta Lake

Unity Catalog

Lakeflow

ADLS Gen2

Key Vault

Git

Job description

About the role

As a Data Engineer, you’ll bring a well-rounded understanding of data-driven practices to support senior team members and stakeholders in enhancing regulatory processes. From uncovering insights through data profiling and analytics to building robust models and engineering solutions, you’ll play a key role in how data is governed, interpreted, and transformed into actionable intelligence.

Key responsibilities of this role are:

• Collaborate within a high-performing Scrum team to deliver BAU work and Project Roadmap milestones that support the Data Science and Analytics Strategy. This includes interpreting requirements, estimating effort, and raising risks and impediments early.
• Interpret business requirements, estimate workloads, prioritise tasks, and manage time in the delivery of quality solutions to meet deadlines.
• Design, build, and maintain scalable data pipelines in Databricks using Python, PySpark, and Spark SQL, primarily within our metadata-driven ingestion and transformation framework.
• Develop and operate Databricks notebooks and jobs on classic compute. Optimize performance and cost through cluster configuration, cluster policies, and Spark and Delta Lake tuning.
• Extend and improve the framework's configuration, patterns, and reusable components so that new data sources can be onboarded quickly and consistently.
• Contribute to the platform's evolution toward Lakeflow declarative pipelines and modern Databricks capabilities.
• Deliver changes through modern CI/CD practices, including source control, automated testing, peer code review, and CAB-approved releases.
• Support, maintain, and upgrade existing workflows, deployment pipelines, and environments across the full application lifecycle.
• Implement and manage data security with Unity Catalog, including catalog, schema, and table permissions, row filters, column masks, service principals, and secrets management.
• Identify, classify, and protect sensitive and personal (PII) data. Apply tagging, masking, and access controls in line with privacy obligations and organizational policy.
• Participate in a rotating incident management roster. This includes troubleshooting and resolving issues, performing root cause analyses, configuring monitoring and alerting, and writing troubleshooting guides.
• Proactively remediate and document operational incidents, project risks, and impediments where possible or otherwise surface these items as appropriate to project managers and senior team members.

NHVR’s Data & Technology teams are currently undertaking growth and considerable transformation, and this is an exciting time to join the organisation.

To learn more, the position description can be found on the NHVR Careers site - https://careers.nhvr.gov.au/jobs/search

About You

You’ll have at least four years of data engineering experience, including significant hands-on work with Databricks or equivalent cloud data platform. A bachelor's degree in a relevant field is desirable. You’ll also be experienced with Agile Scrum, the SDLC, and data lifecycle processes.

Your toolkit for success:

• Hands-on Databricks experience, including notebooks, Jobs/Workflows, cluster configuration on classic compute, and Spark performance tuning.
• Strong Python and PySpark data pipeline development skills, with advanced SQL (Spark SQL and T-SQL or similar).
• Practical experience with Delta Lake, including MERGE and upserts, schema evolution, OPTIMIZE and VACUUM, and partitioning or clustering strategies.
• Experience building or working within metadata-driven or configuration-driven pipeline frameworks.
• Working knowledge of Unity Catalog security and governance, including the securable model, grants and ownership, external locations, storage credentials, and lineage.
• Experience identifying and handling sensitive and PII data, including classification, masking, and least-privilege access.
• Solid data modelling knowledge (relational, dimensional, and medallion architecture) and an understanding of RDBMS concepts.
• Software engineering discipline, including modular and reusable code, unit testing, Git-based workflows, CI/CD, and debugging.
• Experience with Azure data services such as ADLS Gen2, and Key Vault.
• Understanding of batch and streaming data concepts, structured and unstructured data, and data governance

Ideally you will have:

• Experience with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Lakeflow Connect, Auto Loader, and data quality expectations.
• Relevant certifications including Professional or Associate Databricks Data Engineer, and Azure Certification.
• Experience with data visualization tools such as Microsoft PowerBI, Tableau.
• Familiarity with supporting machine learning workloads, such as MLflow.

Why join us?

The National Heavy Vehicle Regulator (NHVR) commenced operations in 2013 as the national entity responsible for regulating all vehicles in Australia over 4.5 tonnes gross vehicle mass. The NHVR minimises the compliance burden on the heavy vehicle transport industry, reduces duplication and inconsistences across state and territory borders, and ultimately for the heavy vehicle business with government in Australia.

We are headquartered in Brisbane and employ more than 1000 people across the ACT, New South Wales, Queensland, South Australia, Tasmania and Victoria.

We offer

• Salary package of $135,995.90 + superannuation
• Monthly RDO and paid leave over Christmas closure.
• Hybrid working environment – up to 2 days from home.
• Up to15.25% superannuation through a co-contribution scheme
• Leave loading 17.5%
• Wellbeing programs – discounted fitness through Fitness Passport, Discount health insurance, flu vaccine, financial programs including discounted banking products, finance information sessions.
• Salary packaging options including novated lease.
• Discounted IT products including Microsoft and free family password security plan.
• Ongoing training and professional development support
• Professional membership

Successful applicants will be required to complete standard pre-employment checks including a national police and work rights check.

We are committed to being an inclusive, diverse and flexible workplace where differences are valued.We welcome applications from people of all backgrounds, experience and abilities.