Data Engineer

HorizonOne Recruitment; Canberra; Hybrid; AUD 120,000 - 150,000; Full time

Benefits offered by this job

Hybrid working across Australia

Contract role with extension option

Job summary

HorizonOne Recruitment is partnering with a specialist government agency to recruit a senior data engineer. The role focuses on building complex data pipelines, turning unstructured data into structured formats, and delivering production-grade ML/NLP pipelines with robust MLOps practices.

Hybrid work across multiple Australian locations is offered, with a long-term contract from October 2026 to June 2027 and a potential 12-month extension.

Qualifications

• Senior data engineer with advanced Python skills and experience building pipelines at scale.
• Hands-on experience with Databricks or similar cloud platforms.
• Track record deploying ML/NLP/generative AI into production with model serving and MLOps practices.

Responsibilities

• Refactor and expand the data ingestion stage for new use cases and structured outputs.
• Automate data ingestion across multiple sources and implement data quality checks.
• Build and maintain model-serving, deployment and API integration capabilities; ensure CI/CD and documentation.

Skills

Advanced Python

Databricks

ML in production

APIs

MLOps

Tools

Databricks

Job description

This is a specialist government agency that provides data, research and analysis on key economic and social themes.

You’ll be working on a significant national policy program, building a standardised, evidence-based framework used to classify and connect information across interconnected sectors.

The Opportunity

This is your chance to build the complex data pipelines behind a national classification challenge.

• Real influence over how the technical build takes shape, not just maintaining someone else’s pipeline
• Led by a people-focused Band 1 with a robust academic background and a clear direction for the project
• Take existing pipelines and AI models into production as a fully operationalised CI/CD capability
• Core stack is Databricks and Python
• Long-term contract, October 2026 to 30 June 2027, with a 12-month extension option

The Role

The key technical priorities for you to deliver in the first 3-6 months are:

• Refactor and expand the data ingestion stage to support new use cases, including turning unstructured data into structured data
• Automate data ingestion across multiple data sources
• Design and implement a data quality and validation layer for large-scale datasets

Additionally, you will be:

• Building and maintaining model-serving, deployment and API integration capabilities
• Applying data governance, metadata, lineage and versioning practices alongside DevOps/MLOps standards – CI/CD, testing and documentation
• Working with data scientists, policy and information management stakeholders to translate agreed definitions and business rules into the pipeline, including integrating outputs with downstream systems

The Ideal Person

You’re a senior data engineer with advanced Python skills, comfortable managing large-scale datasets, automating workflows and building pipelines across cloud or distributed platforms.

You’ve taken machine learning, NLP or generative AI solutions into production, with the data quality, lineage and MLOps practices to back it up.

You will bring the following skills and experience:

• Advanced Python and modern data engineering skills, at scale
• Hands-on experience with Databricks or an equivalent cloud/distributed platform
• A track record taking ML, NLP or generative AI into production – APIs, model serving and MLOps included

Due to the nature of the work, Australian Citizenship and an existing Baseline security clearance is mandatory.

Salary

• This role is being offered as an EL1 equivalent full-time labour hire contract from October 2026 to 30 June 2027, with a 12-month extension option
• There is no set rate for this RFQ. However the value for you coming through myself for this opportunity is that I have extensive experience recruiting into this market, I work with each candidate individually to land on a rate that’s competitive
• Hybrid working, multiple locations across the country

HorizonOne is committed to building a diverse and inclusive workforce and we encourage applications from people of all cultures, capabilities and backgrounds.