Senior Research Crawling Engineer

True Talent Solutions Partners; United States; Remote; USD 160,000 - 250,000; Full time

Benefits offered by this job

Remote work

Competitive salary

Visa sponsorship not available

Job summary

True Talent Solutions Partners in the United States seeks a seasoned engineer to build large-scale web crawling and data acquisition infrastructure for AI research and model development.

You will design high-throughput, fault-tolerant crawlers across diverse domains, manage anti-bot systems, and create pipelines to clean, normalize terabyte-to-petabyte data.

This remote position offers a base salary from $160,000 to $250,000 USD per year; visa sponsorship is not available.

Qualifications

• 2+ years building web crawlers or data pipelines; 4+ years preferred; PhDs may qualify with 2+ years.
• Distributed data pipelines at terabyte scale; high-throughput, fault-tolerant systems.
• Proficiency in Go, Rust, Python, Java or C++, with HTTP & networking knowledge.
• Experience addressing anti-bot systems, rate limits, JS-heavy sites; proxies/IP rotation helpful.
• Familiarity with headless browsers such as Playwright or Puppeteer; cloud or bare-metal infra.
• Data quality eval, NLP pipelines, or ML dataset curation is a plus.

Responsibilities

• Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
• Handle anti-bot systems, rate limits, and dynamic, JavaScript-heavy sites.
• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
• Work with research teams to align data collection and dataset quality with modeling needs.
• Monitor crawl coverage, performance, and data quality, and optimize infrastructure for cost, latency, and reliability.

Skills

Go

Rust

Python

Java

C++

HTTP & Networking

Tools

Playwright

Puppeteer

Job description

About the Role

Build large-scale web crawling and data acquisition infrastructure for AI research and model development. You will work with research teams to create reliable, high-quality datasets from a changing and often adversarial web.

What You’ll Do

• Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.

• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.

• Handle anti-bot systems, rate limits, and dynamic, JavaScript-heavy sites.

• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

• Work with research teams to align data collection and dataset quality with modeling needs.

• Monitor crawl coverage, performance, and data quality, and optimize infrastructure for cost, latency, and reliability.

What We’re Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines, with 4 or more years preferred; candidates with PhDs may qualify with 2 or more years.

• Experience operating distributed data pipelines at terabyte scale and building high-throughput, fault-tolerant data acquisition systems.

• Strong proficiency in Go, Rust, Python, Java, or C++, with deep knowledge of HTTP, networking, and browser behavior.

• Hands-on experience addressing anti-bot systems, rate limits, and JavaScript-heavy websites; experience with proxies, IP rotation, and request orchestration is valuable.

• Familiarity with headless browsers such as Playwright or Puppeteer, and cloud or bare-metal infrastructure.

• Experience with data quality evaluation, NLP pipelines, or machine learning dataset curation is a plus.

Compensation & Benefits

Base salary is $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote. No primary location was specified.