Technical Lead Data Engineering

NHVR External Careers; City of Brisbane; Hybrid; AUD 155,000 - 170,000; Full time

Benefits offered by this job

Hybrid work up to 2 days from home

Wellbeing programs

Professional development support

Salary packaging options

Job summary

NHVR is seeking a Technical Lead, Data Engineering to guide a high‑performing team delivering smarter, more efficient regulatory outcomes. You will lead data architecture, lifecycle management, AI software development, information security, and agile delivery.

You will own the platform architecture, drive modernization with Databricks, and mentor engineers to strengthen governance and data quality across the NHVR Enterprise Data Platform.

Qualifications

• Master’s degree in information technology or computer science related or equivalent.
• Minimum 10 years’ experience with Databricks and Azure cloud platform.

Responsibilities

• Be accountable for data pipeline quality and data platform integrity.
• Evolve platform architecture and roadmap toward Lakehouse and Databricks capabilities.
• Design and improve metadata-driven ingestion and transformation framework.
• Define and enforce engineering standards, patterns, and CI/CD practices.
• Provide architecture guidance to senior stakeholders and manageability of data assets.
• Mentor Data Engineers and grow team capability in Databricks and modern engineering.
• Lead scrum planning, estimate technical effort, and assess risks.
• Act as escalation point for incidents and drive monitoring improvements.
• Represent changes at CAB and ensure release quality and security.
• Remain hands-on delivering complex pipeline components.

Skills

Databricks

Azure cloud

Python

PySpark

SQL

Delta Lake

Data governance

CI/CD

REST APIs

Scrum/Agile

Communication

Education

Master’s degree in IT/CS

Tools

Azure DevOps

Databricks

Job description

About the role

We are now seekingstrategic and hands‑on data leader to guideourhigh‑performing team in delivering smarter, more efficient regulatory outcomeswhilstplayinga key role touplift processes, strengthen governance and drive reliable data solutions.

AsTechnical Lead, Data Engineeringyou willbring expertise across data architecture,datalifecycle management,AIsoftware development, information security, and agile delivery.

Key responsibilities of this role are:

• Be accountable for the quality of data pipeline development and the flow of data on the NHVR Enterprise Data Platform. Quality is measured by data integrity, pipeline performance, uptime, efficient administration, cost, data security, data governance, and architectural integrity.
• Own and evolve the platform architecture and technical roadmap. This includes leading the transition from notebook- and job-based pipelines toward Lakeflow declarative pipelines and other modern Databricks capabilities.
• Own the design and continuous improvement of the metadata-driven ingestion and transformation framework. Ensure it stays scalable, maintainable, well-documented, and consistently applied.
• Define, document, and enforce engineering standards, patterns, and practices, covering coding standards, Delta Lake and medallion design, testing, CI/CD, and observability. Ensure compliance through design reviews and code reviews.
• Provide authoritative advice to Managers, Directors, and other senior staff across the organisation regarding architecture, augmentation, implementation, and manageability of the data on the NHVR Enterprise Data Platform.
• Provide technical guidance, mentoring, and coaching to Data Engineers, including pairing on complex problems and building team capability in Databricks and modern engineering practice.
• Lead technical planning within the Scrum team: break down epics, assess solution options, estimate technical effort, and identify technical risks and dependencies early.
• Act as the technical escalation point for complex incidents, leading root cause analyses and driving lasting improvements to monitoring, alerting, and resilience.
• Represent data engineering changes at CAB and ensure releases meet quality, security, and change management requirements.
• Remain hands-on, delivering complex or high-risk pipeline and framework components alongside the team.

NHVR’sData Science and Analytics Unitare currently undertaking growth and considerable transformation, and this is an exciting time to join the organisation.

To learn more, the position description can be found on the NHVR Careers site - https://careers.nhvr.gov.au/jobs/search

About You

To be successful in this role you will possessa Master’s degree in information technology or computer science related or equivalent. You will have a minimum of 10 years’ experience with demonstrated skills in Databricks and Azure cloud platform (certifications in either / both would be highly regarded)

Your toolkit for success:

• Expert Python and PySpark development skills, with advanced SQL (Spark SQL and T-SQL or similar).
• Deep Databricks expertise, including notebooks, Jobs/Workflows, cluster configuration and policies on classic compute, and advanced Spark performance tuning.
• Advanced Delta Lake knowledge, including MERGE and upsert patterns, schema evolution, OPTIMIZE and VACUUM, liquid clustering and partitioning strategy, and time travel.
• Demonstrated experience designing, building, or owning metadata-driven or configuration-driven pipeline frameworks.
• Demonstrated experience designing and implementing Unity Catalog security and governance models, including grants and ownership, row and column level security, external locations, storage credentials, and lineage.
• Proven experience establishing approaches for identifying, classifying, and protecting sensitive and PII data.
• Strong data architecture and modelling skills, covering relational, dimensional, and medallion architecture, normalisation, schemas, and data profiling.
• Proven code review, software analysis, and design review skills, with a track record of defining and enforcing engineering standards.
• Strong software engineering discipline, including modular design and design patterns, unit and data quality testing (e.g. pytest, data quality frameworks), Git-based workflows, CI/CD with Azure DevOps, and debugging.
• Extensive experience with ELT/ETL pipelines and data integration, including REST APIs, file-based exchange (e.g. SFTP), change data capture, and streaming or event-based patterns.
• Experience leading technical teams in Scrum and Agile methods, with the ability to mentor engineers and influence without direct authority.
• Excellent communication skills, including explaining complex technical matters and trade-offs to senior, non-technical stakeholders

Why join us?

The National Heavy Vehicle Regulator (NHVR) commenced operations in 2013 as the national entity responsible for regulating all vehicles in Australia over 4.5 tonnes gross vehicle mass. The NHVR minimises the compliance burden on the heavy vehicle transport industry, reduces duplication and inconsistences across state and territory borders, and ultimately for the heavy vehicle business with government in Australia.

We are headquartered in Brisbane and employ more than 1000 people across the ACT, New South Wales, Queensland, South Australia, Tasmania and Victoria.

We offer

• A competitive salary package of $166,662.51 per annum + Superannuation.
• Monthly RDO and paid leave over Christmas closure.
• Hybrid working environment – up to 2 days from home.
• Up to15.25% superannuation through a co-contribution scheme
• Leave loading 17.5%
• Wellbeing programs – discounted fitness through Fitness Passport, Discount health insurance, flu vaccine, financial programs including discounted banking products, finance information sessions.
• Salary packaging options including novated lease.
• Discounted IT products including Microsoft and free family password security plan.
• Ongoing training and professional development support
• Professional membership

We are committed to being an inclusive, diverse and flexible workplace where differences are valued.We welcome applications from people of all backgrounds, experience and abilities.