Remote Research Web Crawler Engineer

Wynd Labs

New York (NY)

Hybrid

USD 120,000 - 180,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Wynd Labs is seeking a Research Crawling Engineer to design and operate large-scale web data acquisition systems for research and model development, spanning distributed systems, scraping infrastructure, and data pipelines. The role requires overlapping EST business hours for collaboration.

You will build crawlers, tackle anti-bot and JS-heavy sites, and own end-to-end data pipelines for research and model training, while maintaining data quality and system reliability across scale.

Qualifications

  • Strong programming experience in one or more of Go, Rust, Python, Java, or C++.
  • Experience building web crawlers or large-scale data pipelines.
  • Solid understanding of HTTP, networking, and browser behavior.
  • Familiarity with distributed systems and parallel processing.
  • Experience working with large datasets (TB–PB scale preferred).
  • Ability to debug unstable or adversarial environments.

Responsibilities

  • Build and maintain large-scale web crawlers across diverse domains
  • Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day)
  • Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites
  • Develop pipelines for cleaning, deduplication, filtering, and normalization
  • Construct and maintain datasets for research and model training
  • Monitor crawl performance, coverage, and data quality; iterate quickly
  • Collaborate with research teams to align data collection with modeling needs
  • Optimize infrastructure for cost, latency, and reliability

Skills

Go
Rust
Python
Java
C++

Job description

Wynd Labs is seeking a Research Crawling Engineer to design and operate large-scale web data acquisition systems for research and model development, spanning distributed systems, scraping infrastructure, and data pipelines. The role requires overlapping EST business hours for collaboration.

You will build crawlers, tackle anti-bot and JS-heavy sites, and own end-to-end data pipelines for research and model training, while maintaining data quality and system reliability across scale.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Research Engineer — Large-Scale Web Crawlers
Remote Research Engineer — Large-Scale Web Crawlers

ElevenLabs • United States

Remote
USD 140,000 - 210,000
Remote Research Engineer - Web Crawling & Data Pipelines
Remote Research Engineer - Web Crawling & Data Pipelines

ElevenLabs Inc. • Maine

Hybrid
USD 140,000 - 210,000
Discretionary stipend
Travel stipend
Co-working stipend
+1
Research Crawling Engineer
Research Crawling Engineer

Startup Talents • London (KY)

Remote
USD 150,000 - 225,000
Research Crawling Engineer
Research Crawling Engineer

Wynd Labs • New York (NY)

Hybrid
USD 120,000 - 180,000
Remote Research Crawling Engineer - Scale Data Pipelines
Remote Research Crawling Engineer - Scale Data Pipelines

Startup Talents • London (KY)

Remote
USD 150,000 - 225,000
Remote Data Infrastructure Engineer - Web Crawling Pipelines
Remote Data Infrastructure Engineer - Web Crawling Pipelines

Unknown • United States

Remote
USD 160,000 - 250,000
Fully remote role
Research Engineer - Web Crawlers
Research Engineer - Web Crawlers

ElevenLabs • United States

Remote
USD 140,000 - 210,000
Research Engineer - Web Crawlers
Research Engineer - Web Crawlers

ElevenLabs Inc. • Maine

Remote
USD 140,000 - 210,000
Discretionary stipend
Travel stipend
Co-working stipend
+1
AI-Driven Web Data Crawling Engineer
AI-Driven Web Data Crawling Engineer

Rubick.AI • Oregon

Hybrid
USD 90,000 - 150,000
Remote Python Engineer — Web Crawlers & Scrapy
Remote Python Engineer — Web Crawlers & Scrapy

ZYTE • United States

Remote
USD 100,000 - 140,000
35 days paid holidays
Flexible remote work