Distributed Web Crawler Engineer — Remote

UMAI Consultancy; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

ElevenLabs is seeking a Research Engineer to join the research team, focused on large-scale web crawling for our frontier AI models. You will own the crawling systems that source world-class data from the open web, enabling high-quality training data for our models.

You will thrive if you enjoy building distributed pipelines, solving deduplication and recrawl challenges, and creating tooling for researchers to access newly crawled data quickly and reliably.

Qualifications

• We want hands-on experience building and scaling web crawlers or scraping systems, ideally for ML training data.
• Strong engineering skills in distributed systems at scale (Kubernetes, queue-based architectures, or custom pipelines processing billions of documents).
• The capacity to evaluate data quality, coverage, and compliance, and to build tooling to measure it.

Responsibilities

• Build and operate large-scale distributed web crawlers that fetch, extract, and index data from billions of pages.
• Tackle crawling challenges such as content extraction from messy HTML, deduplication, recrawl strategies, and politeness/rate limits.
• Design pipelines to locate high-value sources (audio, video, multilingual content) and convert them into training-ready data.
• Create tooling and infrastructure for researchers to request, monitor, and explore crawled data quickly and reliably.

Skills

Web crawling

Distributed systems

Quality & compliance

Tools

Kubernetes

Message queues

Job description

ElevenLabs is seeking a Research Engineer to join the research team, focused on large-scale web crawling for our frontier AI models. You will own the crawling systems that source world-class data from the open web, enabling high-quality training data for our models.

You will thrive if you enjoy building distributed pipelines, solving deduplication and recrawl challenges, and creating tooling for researchers to access newly crawled data quickly and reliably.