Lead Data Engineer

Clicks IT Recruitment; City of Melbourne; On-site; AUD 180,000 - 230,000; Full time

Job summary

Clicks IT Recruitment is seeking a Lead Data Engineer in Australia to design, build and maintain scalable data and AI infrastructure supporting ML, NLP and Generative AI workloads. You will work within a multidisciplinary team of data scientists and policy experts across nationwide operations.

The role requires experience with scalable data pipelines, cloud-based environments and advanced Python development, plus a Baseline security clearance to apply.

Qualifications

• Experience designing, building and maintaining scalable data pipelines and data platforms.
• Strong expertise in data ingestion, transformation, automation and bulk processing workflows.
• Experience working with cloud-based and distributed data environments.
• Advanced Python development skills.
• Proven experience deploying ML, NLP and/or Generative AI solutions into production.
• Experience supporting end-to-end delivery of AI-enabled products and services.

Responsibilities

• Designing, developing and maintaining scalable data ingestion, transformation and processing pipelines for structured and unstructured data sources.
• Applying modern data engineering practices to support collection, preparation, cleansing, validation and integration of skills-related datasets.
• Developing automated data quality, testing and validation capabilities that support reliable and reproducible analytical outcomes.
• Designing and implementing repeatable validation pipelines that support the assessment, monitoring and quality assurance of AI and NLP model outputs.
• Collaborating with Data Scientists to operationalise machine learning, NLP and Generative AI solutions within scalable production environments.
• Developing and maintaining model-serving, deployment and API integration capabilities that support analytical products and downstream systems.
• Implementing data governance, metadata management, lineage tracking and versioning practices to support transparency, traceability and reproducibility.
• Developing and maintaining cloud-based and distributed data processing environments capable of supporting large-scale data and AI workloads.

Skills

Data pipelines

Python

Cloud platforms

ML/AI deployment

Data governance

Automation / ETL

NLP / Generative AI

Job description

Lead Data Engineer

QLD, WA, ACT, VIC, NSW, NT, SA, TAS

Until June 2027 with 12 months extension

Baseline Clearance Required

We are seeking an experienced Lead Data Engineer to play a key role in the development of a nationally significant AI and data capability focused on workforce, education and skills intelligence. Working within a multidisciplinary team of Data Scientists, Information Management specialists and policy experts, you will design, build and maintain scalable data and AI infrastructure that supports advanced analytics, machine learning, NLP and Generative AI solutions.

Key duties and responsibilities as a Lead Data Engineer:

• Designing, developing and maintaining scalable data ingestion, transformation and processing pipelines for structured and unstructured data sources.
• Applying modern data engineering practices to support the collection, preparation, cleansing, validation and integration of skills-related datasets.
• Developing automated data quality, testing and validation capabilities that support reliable and reproducible analytical outcomes.
• Designing and implementing repeatable validation pipelines that support the assessment, monitoring and quality assurance of AI and NLP model outputs.
• Collaborating with Data Scientists to operationalise machine learning, NLP and Generative AI solutions within scalable production environments.
• Developing and maintaining model-serving, deployment and API integration capabilities that support analytical products and downstream systems.
• Implementing data governance, metadata management, lineage tracking and versioning practices to support transparency, traceability and reproducibility.
• Developing and maintaining cloud-based and distributed data processing environments capable of supporting large-scale data and AI workloads.

Skills & Experience Essential

• Demonstrated experience designing, building and maintaining scalable data pipelines and data platforms.
• Strong expertise in data ingestion, transformation, automation and bulk processing workflows.
• Experience working with cloud-based and distributed data environments.
• Advanced Python development skills.
• Proven experience deploying Machine Learning, NLP and/or Generative AI solutions into production environments.
• Experience supporting end-to-end delivery of AI-enabled products and services.

Please note you will need to have an existing AGSVA Baseline security Clearance to apply for this role

At Clicks we embrace diversity, inclusion and equal opportunity. First Nations people are encouraged to apply.

We provide reasonable adjustments, including alternate formats to the recruitment process for individuals with a disability. If you require an adjustment to be made during the recruitment process, please call 1300 254 257 or [e-mail]