Senior Petabyte-Scale Web Crawling Engineer

Thinking Machines Lab Inc.; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. is seeking a Software Engineer to own and scale our web-crawling systems, from distributed collection to filtering and data preparation for pretraining.

You will build the crawler, the scale-out infrastructure, and the data pipelines that convert raw crawls into usable datasets. You’ll collaborate with pretraining and data teams to align crawler outputs with model quality goals, while maintaining production-grade reliability for petabyte-scale operations.

Qualifications

• 8+ years designing, building, and scaling web crawlers or large-scale data-acquisition systems.
• Experience owning crawler infrastructure at internet scale.
• Proficiency in Python, Go, or Rust with distributed systems experience.
• Knowledge of robots.txt, rate limiting, and data licensing.

Responsibilities

• Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data.
• Build pipelines for large-scale extraction, deduplication, and data quality filtering.
• Build specialized crawlers for high-value or hard-to-reach data sources.
• Work with the pretraining team to understand how changes in crawled data affect model performance.
• Improve reliability and efficiency of crawling and ingestion infrastructure at petabyte scale.
• Help set technical direction for this area as it grows, and bring other engineers up to speed on what you've learned.

Skills

Distributed systems

Web crawlers

Python/Go/Rust

Robots.txt/licensing knowledge

Job description

Thinking Machines Lab Inc. is seeking a Software Engineer to own and scale our web-crawling systems, from distributed collection to filtering and data preparation for pretraining.

You will build the crawler, the scale-out infrastructure, and the data pipelines that convert raw crawls into usable datasets. You’ll collaborate with pretraining and data teams to align crawler outputs with model quality goals, while maintaining production-grade reliability for petabyte-scale operations.