Remote: Distributed Web Crawler Engineer

Jackalope Digital LLC; United Kingdom; Remote; GBP 85,000 - 140,000; Full time

Benefits offered by this job

Innovative culture

Growth paths

Learning & development

Social travel

Annual company offsite

Co-working

Job summary

ElevenLabs in the United Kingdom is seeking a Research Engineer to join the research team, focusing on large-scale web crawling for frontier AI models. You will own the crawling systems that source world-class data from the open web and thrive on building scalable, reliable tooling.

You’ll tackle challenges like content extraction from messy HTML, deduplication at web scale, freshness and recrawl strategies, and politeness controls, while designing pipelines to find high-value data sources

Qualifications

• Hands-on experience building and scaling web crawlers or scraping systems.
• Strong engineering skills in distributed systems at scale (e.g., Kubernetes, queue-based architectures, or custom pipelines processing billions of documents).
• The capacity to autonomously evaluate the quality, coverage, and compliance of crawled data, and to build the tooling to measure it.

Responsibilities

• Build and operate large-scale distributed web crawlers for ML data.
• Address crawling challenges such as content extraction from messy HTML, deduplication at web scale, freshness and recrawl strategies, and politeness and rate-limit handling.
• Design targeted crawling pipelines that find high-value data sources, including audio, video, and multilingual content, and turn them into clean training-ready datasets.
• Create tooling and infrastructure that lets researchers request, monitor, and explore newly crawled web data quickly and reliably.

Job description

ElevenLabs in the United Kingdom is seeking a Research Engineer to join the research team, focusing on large-scale web crawling for frontier AI models. You will own the crawling systems that source world-class data from the open web and thrive on building scalable, reliable tooling.

You’ll tackle challenges like content extraction from messy HTML, deduplication at web scale, freshness and recrawl strategies, and politeness controls, while designing pipelines to find high-value data sources