Senior Web Crawling & Data Infrastructure Engineer

Bison; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Paid parental leave

Relocation support as needed

Job summary

Thinking Machines is hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep. This role is based in San Francisco, CA or New York, NY, with compensation in the range of $350,000-$475,000 USD annually.

You will design scalable crawlers, pipelines for extraction and deduplication, and work with cross‑functional teams to improve data quality and model outcomes.

Qualifications

• 8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data‑acquisition systems.
• A track record of owning crawler or data‑acquisition infrastructure at internet scale.
• Strong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems.
• Working knowledge of the practical and legal considerations of large-scale web data collection (robots.txt, rate limiting, licensing).

Responsibilities

• Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data.
• Build pipelines for large-scale extraction, deduplication, and data quality filtering.
• Build specialized crawlers for high-value or hard-to-reach data sources.
• Work with the pretraining team to understand how changes in crawled data affect model performance.
• Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale.
• Help set technical direction for this area as it grows, and bring other engineers up to speed on what you've learned.

Skills

Web crawlers / data acquisition

Distributed systems

Python / Go / Rust

Robots.txt / data licensing awareness

Tools

Python

Go

Rust

Job description

Thinking Machines is hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep. This role is based in San Francisco, CA or New York, NY, with compensation in the range of $350,000-$475,000 USD annually.

You will design scalable crawlers, pipelines for extraction and deduplication, and work with cross‑functional teams to improve data quality and model outcomes.