Web Crawler Engineer - Build Google-scale Crawling & Search

Exa; Zürich; On-site; CHF 120,000 - 160,000; Full time

Benefits offered by this job

Premium healthcare benefits (medical,

Job summary

Exa in Zürich is seeking a Web Crawler Engineer to design and scale a Google‑scale crawler and related distributed systems. You will build high‑performance infrastructure to crawl the web, process massive page loads, and optimize coverage and quality of results.

You should be proficient with C++/Rust, TypeScript, Playwright, and Chrome DevTools Protocol, and you will focus on maximizing crawl efficiency and system performance.

Qualifications

• Extensive experience building and scaling web crawlers.
• Experience with high-performance languages (C++, Rust).
• Familiar with TypeScript, Playwright, CDP (Chrome DevTools Protocol).
• Comfortable optimizing systems to a high degree.

Responsibilities

• Build a distributed crawler that can handle 100M+ pages per day.
• Optimize crawl politeness and rate limiting across thousands of domains.
• Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures.
• Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency.

Skills

Web crawling

C++

Rust

TypeScript

Playwright

CDP (Chrome DevTools Protocol)

Performance optimization

Tools

Playwright

CDP

Job description

Exa in Zürich is seeking a Web Crawler Engineer to design and scale a Google‑scale crawler and related distributed systems. You will build high‑performance infrastructure to crawl the web, process massive page loads, and optimize coverage and quality of results.

You should be proficient with C++/Rust, TypeScript, Playwright, and Chrome DevTools Protocol, and you will focus on maximizing crawl efficiency and system performance.