Frontier Data Engineer — AI Pipelines & Crawling | Equity

Maincode; Melbourne; Híbrido; AUD 120 000 - 180 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Equity

Superannuation

Resumo da oferta

Maincode is seeking a data-focused engineer to build and scale web crawlers and data pipelines for training frontier AI models. You will tackle diverse sources including legal and regulatory content, ensuring high-quality, compliant data for model training.

The role requires hands-on experience with Python and distributed data processing, plus expertise in crawler challenges and in integrating LLMs in data flows. Melbourne-based, with hybrid/work-from-anywhere options across Australia.

Qualificações

• Experience building and running large-scale web crawling or scraping systems in production.
• Proficiency in Python; hands-on with Go, Rust, or C++ used in distributed data processing.
• Experience with distributed data processing frameworks (Spark, Ray, Beam) and petabyte-scale storage.
• Hands-on knowledge of crawling challenges (JS-heavy sites, headless browsers, rate limiting, robots.txt).
• Experience using LLMs and AI agents inside data pipelines for classification and data quality scoring.

Responsabilidades

• Build, run, and scale crawlers and scrapers for web-scale collection across high-value sources such as legislation and government publications.
• Develop robust pipelines for extraction, filtering, cleaning, deduplication, and formatting for model training inputs.
• Create agentic pipelines where AI agents generate, annotate, critique, and verify data at scale with evaluations and spot checks.
• Turn market requirements into dataset specifications, data mixtures, and measurable evaluations.
• Collaborate with core training team to iterate on data mixtures, evaluate data quality, and respond to model performance feedback.
• Track provenance, licensing, privacy, and security for every dataset shipped.

Conhecimentos

Large-scale web crawling

Python

Go/Rust/C++

Distributed data processing

Spark/Ray/Beam

Petabyte-scale storage

Headless browsers/JS crawling

LLMs in data pipelines

Data privacy & licensing

Ferramentas

Spark

Ray

Beam

Descrição da oferta de emprego

Maincode is seeking a data-focused engineer to build and scale web crawlers and data pipelines for training frontier AI models. You will tackle diverse sources including legal and regulatory content, ensuring high-quality, compliant data for model training.

The role requires hands-on experience with Python and distributed data processing, plus expertise in crawler challenges and in integrating LLMs in data flows. Melbourne-based, with hybrid/work-from-anywhere options across Australia.