Remote Senior Web Crawler Architect for AI Data

True Talent Solutions Partners; United States; Remote; USD 160,000 - 250,000; Full time

Benefits offered by this job

Remote work

Competitive salary

Visa sponsorship not available

Job summary

True Talent Solutions Partners in the United States seeks a seasoned engineer to build large-scale web crawling and data acquisition infrastructure for AI research and model development.

You will design high-throughput, fault-tolerant crawlers across diverse domains, manage anti-bot systems, and create pipelines to clean, normalize terabyte-to-petabyte data.

This remote position offers a base salary from $160,000 to $250,000 USD per year; visa sponsorship is not available.

Qualifications

• 2+ years building web crawlers or data pipelines; 4+ years preferred; PhDs may qualify with 2+ years.
• Distributed data pipelines at terabyte scale; high-throughput, fault-tolerant systems.
• Proficiency in Go, Rust, Python, Java or C++, with HTTP & networking knowledge.
• Experience addressing anti-bot systems, rate limits, JS-heavy sites; proxies/IP rotation helpful.
• Familiarity with headless browsers such as Playwright or Puppeteer; cloud or bare-metal infra.
• Data quality eval, NLP pipelines, or ML dataset curation is a plus.

Responsibilities

• Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
• Handle anti-bot systems, rate limits, and dynamic, JavaScript-heavy sites.
• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
• Work with research teams to align data collection and dataset quality with modeling needs.
• Monitor crawl coverage, performance, and data quality, and optimize infrastructure for cost, latency, and reliability.

Skills

Go

Rust

Python

Java

C++

HTTP & Networking

Tools

Playwright

Puppeteer

Job description

True Talent Solutions Partners in the United States seeks a seasoned engineer to build large-scale web crawling and data acquisition infrastructure for AI research and model development.

You will design high-throughput, fault-tolerant crawlers across diverse domains, manage anti-bot systems, and create pipelines to clean, normalize terabyte-to-petabyte data.

This remote position offers a base salary from $160,000 to $250,000 USD per year; visa sponsorship is not available.