Frontier AI Data Engineer: Crawling & Pipelines (Equity)

MAI Agents; Melbourne; Hybrid; AUD 110,000 - 190,000; Full time

Benefits offered by this job

Equity

Superannuation

Hybrid/On-site across Australia

Job summary

Maincode is hiring a data-focused engineer to scale web crawlers and data pipelines for Matilda, our Australian-made AI stack. You will build crawlers, ensure data quality, and collaborate with the core training team across Australia.

Own end-to-end data collection, provenance, licensing, privacy, and dataset readiness for model training. Melbourne-based role with hybrid/on-site options across Australia; remuneration negotiable plus equity.

Qualifications

• Building, scaling, and running large-scale web crawling or scraping systems.
• Experience with Python and at least one of Go, Rust, or C++.
• Distributed data processing and petabyte-scale storage experience.
• Hands-on crawling with JavaScript-heavy sites and headless browsers.
• Experience using LLMs inside data pipelines for classification or data generation.
• Bias for shipping: deliver working pipelines quickly and reliably.

Responsibilities

• Build, run, and scale crawlers and scrapers for web-scale collection.
• Develop pipelines for extraction, filtering, cleaning, deduplication, formatting.
• Create agentic pipelines where AI agents annotate and verify data at scale.
• Turn market requirements into dataset specifications and evaluations.
• Collaborate with training team to iterate data mixtures and respond to feedback.
• Track provenance, licensing, privacy, and security for datasets.

Skills

Web crawling

Python

Go

Spark

Headless browsers

Robots.txt

LLMs in pipelines

Distributed processing

Tools

Spark

Ray

Beam

C++

Job description

Maincode is hiring a data-focused engineer to scale web crawlers and data pipelines for Matilda, our Australian-made AI stack. You will build crawlers, ensure data quality, and collaborate with the core training team across Australia.

Own end-to-end data collection, provenance, licensing, privacy, and dataset readiness for model training. Melbourne-based role with hybrid/on-site options across Australia; remuneration negotiable plus equity.