Head of Data for Frontier AI Systems

MAI Agents; Australia; Hybrid; AUD 150,000 - 190,000; Full time

Job summary

Maincode seeks a data-engineering-focused engineer to build and scale datasets for frontier models. You will own end-to-end data pipelines from web crawlers to filtering, deduplication, and agent-driven data generation for training sets.

You will work with the core training team to iterate on data mixtures, evaluate quality, and implement robust pipelines that respect licensing, privacy, and governance across Australia.

Qualifications

• Experience building, scaling, and running large-scale web crawling or scraping systems in production.
• Strong Python engineering with distributed data processing and large storage.
• Hands-on crawling with JavaScript-heavy sites, headless browsers, and rate-limiting constraints.
• Experience using LLMs and AI agents in data pipelines for classification, extraction, or quality scoring.
• Bias for shipping: get a working pipeline running in days and make it reliable.
• Experience leading the technical direction of a project.

Responsibilities

• Build, run, and scale crawlers and scrapers for web-scale collection, including legislation, regulator guidance, government publications, local media, and forums.
• Develop robust pipelines for extraction, filtering, cleaning, deduplication, and formatting for model training inputs.
• Build agentic pipelines where AI agents generate, annotate, critique, and verify data at scale with spot checks.
• Translate market and regulatory requirements into dataset specifications, data mixtures, and evaluations.
• Collaborate with the core training team to iterate on data mixtures and respond to model feedback.
• Track provenance, licensing, privacy, and security for every dataset shipped.

Skills

Web crawling

Python

Go

Rust

C++

Distributed data processing

Petabyte-scale storage

LLMs in data pipelines

Quality assurance

Tools

Spark

Ray

Beam

Job description

Maincode seeks a data-engineering-focused engineer to build and scale datasets for frontier models. You will own end-to-end data pipelines from web crawlers to filtering, deduplication, and agent-driven data generation for training sets.

You will work with the core training team to iterate on data mixtures, evaluate quality, and implement robust pipelines that respect licensing, privacy, and governance across Australia.