Petabyte-Scale Web Crawler & Ingestion Engineer

Thinking Machines Lab; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental and vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. is hiring a Software Engineer to design and own web‑crawling systems, from distributed collection at internet scale through filtering and data selection for pretraining data.

You will implement production crawlers and ingestion pipelines, collaborating with the pretraining and data teams to understand how crawled data affects model performance. This is an engineering role focused on scalable data infrastructure.

Qualifications

• 8+ years building and scaling web crawlers or large-scale data acquisition systems.
• Owned crawler or data-acquisition infrastructure at internet scale.
• Strong software engineering in Python/Go/Rust with distributed systems experience.
• Knowledge of robots.txt, rate limiting, licensing for large-scale data collection.

Responsibilities

• Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data.
• Build pipelines for large-scale extraction, deduplication, and data quality filtering.
• Build specialized crawlers for high-value or hard-to-reach data sources.
• Work with the pretraining team to understand data impact on model performance.
• Improve reliability and efficiency of crawling and ingestion infra at petabyte scale.
• Help set technical direction for this area and mentor other engineers.

Skills

Web crawler design

Distributed systems

Python/Go/Rust

Licensing knowledge

Tools

Python

Go

Rust

Job description

Thinking Machines Lab Inc. is hiring a Software Engineer to design and own web‑crawling systems, from distributed collection at internet scale through filtering and data selection for pretraining data.

You will implement production crawlers and ingestion pipelines, collaborating with the pretraining and data teams to understand how crawled data affects model performance. This is an engineering role focused on scalable data infrastructure.