Remote Research Crawling Engineer - Scale Data Pipelines

Startup Talents; London (KY); On-site; USD 150,000 - 225,000; Full time

Job summary

Startup Talents is hiring a Research Crawling Engineer to join their team in London, United States. This full remote position (USA/EU) entails designing large-scale web data acquisition systems optimized for research and AI model development. Responsibilities include developing fault-tolerant data collection systems, collaborating with research teams, and ensuring data quality. The position offers a competitive salary ranging from $150k to $225k based on experience, along with an equity package. Ideal candidates must possess strong programming skills and experience with web crawlers.

Qualifications

• Strong programming experience in one or more of Go, Rust, Python, Java, or C++.
• Experience designing high-throughput, fault-tolerant systems for data collection.
• Familiarity with NLP pipelines or dataset curation for ML is a plus.

Responsibilities

• Design high-throughput, fault-tolerant systems for data collection.
• Collaborate with research teams to align data collection with modeling needs.
• Monitor crawl performance, coverage, and data quality; iterate quickly.

Skills

Strong programming experience in one or more of: Go, Rust, Python, Java, or C++

Experience building and maintaining large-scale web crawlers or data pipelines

Experience handling anti-bot systems, rate limits, and dynamic/JS-heavy sites

Familiarity with distributed systems and parallel processing

Ability to debug unstable or adversarial environments

Job description

Startup Talents is hiring a Research Crawling Engineer to join their team in London, United States. This full remote position (USA/EU) entails designing large-scale web data acquisition systems optimized for research and AI model development. Responsibilities include developing fault-tolerant data collection systems, collaborating with research teams, and ensuring data quality. The position offers a competitive salary ranging from $150k to $225k based on experience, along with an equity package. Ideal candidates must possess strong programming skills and experience with web crawlers.