Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Exa in Singapore is hiring a Web Crawler engineer to help build Google-scale crawling for AI agents. You’ll work on distributed crawlers, high-performance languages like C++ or Rust, and modern tooling to crawl the web at massive scale.
You’ll design systems for dynamic content, implement polite crawling, and optimize scheduling for maximum coverage. This is an in-person role in Singapore with international sponsorship and benefits.
_ We raised a $250M Series C to build the search engine for AIs. Led by a16z, with existing investors Benchmark, Lightspeed, and YC doubling down, the round brings Exa’s valuation to $2.2 billion. _ Read more _
Exa is building a search engine from scratch to serve every AI agent. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to process it, and design super high performant vector databases in rust to search over it. If you like compute, we also own a $5M H200 GPU cluster (and soon 5x’ing that) and regularly spin up batchjobs with tens of thousands of machines.
As a Web Crawler engineer, you’d be responsible for crawling the entire web. Basically build Google-scale crawling!
You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
You have experience with some high performance language (C++, Rust, etc.)
You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
You’re comfortable optimizing a system to an exceptional degree
You care about the problem of finding high quality knowledge and recognize how important this is for the world
Build a distributed crawler that can handle 100M+ pages per day
Optimize crawl politeness and rate limiting across thousands of domains
Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
This is an in-person opportunity in Singapore. We’re happy to sponsor international candidates.
In addition to premium healthcare benefits (medical, dental, vision), we also offer fertility benefits and a monthly wellness stipend to all of our employees.
SGD90K – SGD300K Offers Equity