Technical Lead Data Engineering

National Heavy Vehicle Regulator; City of Brisbane; Hybrid; AUD 150,000 - 183,000; Full time

Benefits offered by this job

Hybrid working environment

Competitive salary package

Superannuation

Wellbeing programs

Professional development

Job summary

National Heavy Vehicle Regulator (NHVR) seeks a strategic and hands-on Technical Lead, Data Engineering to guide a high-performing team in delivering smarter regulatory outcomes. You will own the data platform architecture, modernize pipelines with Databricks, and ensure data governance and security across the NHVR Enterprise Data Platform.

You will mentor engineers, drive the technical roadmap, and collaborate with senior stakeholders to uplift capabilities while delivering complex data

Qualifications

• Master’s degree in IT or computer science, or equivalent.
• Minimum 10 years’ experience with Databricks and Azure.
• Certifications in Databricks and/or Azure highly regarded.

Responsibilities

• Lead data engineering with Databricks and Azure cloud platform.
• Own platform architecture and technical roadmap.
• Mentor Data Engineers and lead agile Scrum teams.
• Ensure data quality, security, and governance across pipelines.

Skills

Python

PySpark

Databricks

Delta Lake

Unity Catalog

Data architecture

CI/CD

REST APIs

Scrum / Agile leadership

Azure

SQL

Education

Master’s degree in information technology or computer science

Equivalent experience

Tools

Azure cloud platform

Databricks notebooks & Jobs

Git-based workflows

Job description

National Heavy Vehicle Regulator – Brisbane QLD

We are now seekingstrategic and hands‑on data leader to guideourhigh‑performing team in delivering smarter, more efficient regulatory outcomeswhilstplayinga key role touplift processes, strengthen governance and drive reliable data solutions.

AsTechnical Lead, Data Engineeringyou willbring expertise across data architecture,datalifecycle management,AIsoftware development, information security, and agile delivery. Key responsibilities of this role are:

• Be accountable for the quality of data pipeline development and the flow of data on the NHVR Enterprise Data Platform. Quality is measured by data integrity, pipeline performance, uptime, efficient administration, cost, data security, data governance, and architectural integrity.
• Own and evolve the platform architecture and technical roadmap. This includes leading the transition from notebook- and job-based pipelines toward Lakeflow declarative pipelines and other modern Databricks capabilities.
• Own the design and continuous improvement of the metadata-driven ingestion and transformation framework. Ensure it stays scalable, maintainable, well-documented, and consistently applied.
• Define, document, and enforce engineering standards, patterns, and practices, covering coding standards, Delta Lake and medallion design, testing, CI/CD, and observability. Ensure compliance through design reviews and code reviews.
• Provide authoritative advice to Managers, Directors, and other senior staff across the organisation regarding architecture, augmentation, implementation, and manageability of the data on the NHVR Enterprise Data Platform.
• Provide technical guidance, mentoring, and coaching to Data Engineers, including pairing on complex problems and building team capability in Databricks and modern engineering practice.
• Lead technical planning within the Scrum team: break down epics, assess solution options, estimate technical effort, and identify technical risks and dependencies early.
• Act as the technical escalation point for complex incidents, leading root cause analyses and driving lasting improvements to monitoring, alerting, and resilience.
• Represent data engineering changes at CAB and ensure releases meet quality, security, and change management requirements.
• Remain hands‑on, delivering complex or high‑risk pipeline and framework components alongside the team.

NHVR’sData Science and Analytics Unitare currently undertaking growth and considerable transformation, and this is an exciting time to join the organisation.

To learn more, the position description can be found on the NHVR Careers site -https://careers.nhvr.gov.au/jobs/search

About You

To be successful in this role you will possessa Master’s degree in information technology or computer science related or equivalent. You will have a minimum of 10 years’ experience with demonstrated skills in Databricks and Azure cloud platform (certifications in either / both would be highly regarded)

Your toolkit for success:

• Expert Python and PySpark development skills, with advanced SQL (Spark SQL and T-SQL or similar).
• Deep Databricks expertise, including notebooks, Jobs/Workflows, cluster configuration and policies on classic compute, and advanced Spark performance tuning.
• Advanced Delta Lake knowledge, including MERGE and upsert patterns, schema evolution, OPTIMIZE and VACUUM, liquid clustering and partitioning strategy, and time travel.
• Demonstrated experience designing, building, or owning metadata-driven or configuration-driven pipeline frameworks.
• Demonstrated experience designing and implementing Unity Catalog security and governance models, including grants and ownership, row and column level security, external locations, storage credentials, and lineage.
• Proven experience establishing approaches for identifying, classifying, and protecting sensitive and PII data.
• Strong data architecture and modelling skills, covering relational, dimensional, and medallion architecture, normalisation, schemas, and data profiling.
• Proven code review, software analysis, and design review skills, with a track record of defining and enforcing engineering standards.
• Strong software engineering discipline, including modular design and design patterns, unit and data quality testing (e.g. pytest, data quality frameworks), Git-based workflows, CI/CD with Azure DevOps, and debugging.
• Extensive experience with ELT/ETL pipelines and data integration, including REST APIs, file-based exchange (e.g. SFTP), change data capture, and streaming or event-based patterns.
• Experience leading technical teams in Scrum and Agile methods, with the ability to mentor engineers and influence without direct authority.
• Excellent communication skills, including explaining complex technical matters and trade-offs to senior, non-technical stakeholders

The National Heavy Vehicle Regulator (NHVR) commenced operations in 2013 as the national entity responsible for regulating all vehicles in Australia over 4.5 tonnes gross vehicle mass. The NHVR minimises the compliance burden on the heavy vehicle transport industry, reduce duplication and inconsistences across state and territory borders, and ultimately for the heavy vehicle business with government in Australia.

We are headquartered in Brisbane and employ more than 1000 people across the ACT, New South Wales, Queensland, South Australia, Tasmania and Victoria.

We offer-

• A competitive salary package of $166,662.51 per annum + Superannuation.
• Monthly RDO and paid leave over Christmas closure.
• Hybrid working environment – up to 2 days from home.
• Up to15.25% superannuation through a co-contribution scheme
• Leave loading 17.5%
• Wellbeing programs – discounted fitness through Fitness Passport, Discount health insurance, flu vaccine, financial programs including discounted banking products, finance information sessions.
• Salary packaging options including novated lease.
• Discounted IT products including Microsoft and free family password security plan.
• Ongoing training and professional development support
• Professional membership

If you have any questions, please don’t hesitate to contact the NHVR recruitment team on ***********@nhvr.gov.au.

Successful applicants will be required to complete standard pre-employment checks including a national police and work rights check.

Please Note

We will not require any support from Recruitment Agencies at this time and kindly request no contact is made. Only direct applications will be considered.

We are committed to being an inclusive, diverse and flexible workplace where differences are valued.We welcome applications from people of all backgrounds, experience and abilities.