Software Engineer, Web Crawling

Exa

Singapore

On-site

SGD 90,000 - 300,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Premium healthcare
Fertility benefits
Monthly wellness stipend

Job summary

Exa in Singapore is hiring a Web Crawler engineer to help build Google-scale crawling for AI agents. You’ll work on distributed crawlers, high-performance languages like C++ or Rust, and modern tooling to crawl the web at massive scale.

You’ll design systems for dynamic content, implement polite crawling, and optimize scheduling for maximum coverage. This is an in-person role in Singapore with international sponsorship and benefits.

Qualifications

  • You have extensive experience building and scaling web crawlers or can ramp up quickly.
  • You have experience with high-performance languages like C++ or Rust.
  • You are familiar with TypeScript and modern web tooling such as Playwright and CDP.

Responsibilities

  • Build a distributed crawler capable of 100M+ pages per day.
  • Optimize crawl politeness and rate limiting across thousands of domains.
  • Design systems to detect dynamic content, JS rendering, and anti-bot measures.
  • Create intelligent crawl scheduling for maximum coverage efficiency.

Skills

Web crawling
C++
Rust
TypeScript
Playwright
CDP (Chrome DevTools Protocol)
Performance optimization

Tools

Playwright
CDP

Job description

_ We raised a $250M Series C to build the search engine for AIs. Led by a16z, with existing investors Benchmark, Lightspeed, and YC doubling down, the round brings Exa’s valuation to $2.2 billion. _ Read more _

Exa is building a search engine from scratch to serve every AI agent. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to process it, and design super high performant vector databases in rust to search over it. If you like compute, we also own a $5M H200 GPU cluster (and soon 5x’ing that) and regularly spin up batchjobs with tens of thousands of machines.

As a Web Crawler engineer, you’d be responsible for crawling the entire web. Basically build Google-scale crawling!

Who You Are
  • You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly

  • You have experience with some high performance language (C++, Rust, etc.)

  • You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)

  • You’re comfortable optimizing a system to an exceptional degree

  • You care about the problem of finding high quality knowledge and recognize how important this is for the world

What You Could Do
  • Build a distributed crawler that can handle 100M+ pages per day

  • Optimize crawl politeness and rate limiting across thousands of domains

  • Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures

  • Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

This is an in-person opportunity in Singapore. We’re happy to sponsor international candidates.

In addition to premium healthcare benefits (medical, dental, vision), we also offer fertility benefits and a monthly wellness stipend to all of our employees.

Compensation

SGD90K – SGD300K Offers Equity

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Web Crawler Engineer — Build Google-scale Crawling + Equity
Web Crawler Engineer — Build Google-scale Crawling + Equity

Exa • Singapore

On-site
SGD 90,000 - 300,000
Premium healthcare
Fertility benefits
Monthly wellness stipend
Software Engineer, Backend
Software Engineer, Backend

Exa • Singapore

On-site
SGD 75,000 - 100,000
Premium healthcare benefits
16 weeks fully paid parental leave
Monthly wellness stipend
Forward Deployed Engineer, Singapore
Forward Deployed Engineer, Singapore

Exa • Singapore

On-site
SGD 150,000 - 210,000
Premium healthcare (medical)
Fertility benefits
Monthly wellness stipend
Software Engineer (AI/ML)
Software Engineer (AI/ML)

Hyphen Partners • Singapore

On-site
SGD 110,000 - 160,000
Senior Search Engine Backend Engineer
Senior Search Engine Backend Engineer

PatSnap • Singapore

On-site
SGD 120,000 - 180,000
Software Engineer - AI Search Infra & Low-Latency
Software Engineer - AI Search Infra & Low-Latency

Hyphen Partners • Singapore

On-site
SGD 110,000 - 160,000
Software Engineer (AI)
Software Engineer (AI)

ScienTec Consulting Pte. Ltd. • Singapore

Hybrid
SGD 107,000 - 128,000
Hybrid work arrangements
AWS
Salary up to S$10,500
Search Engine Evaluator - English (Singapore)
Search Engine Evaluator - English (Singapore)

RWS • Singapore

Remote
SGD 13,000 - 19,000
Flexible home-based work
Work-life balance
Potential side-hustle
+2
Fullstack Software Engineer
Fullstack Software Engineer

PatSnap • Singapore

On-site
SGD 140,000 - 210,000
Competitive compensation
Global teamwork opportunities
Career development paths
+1
Technical Recruiter - APAC
Technical Recruiter - APAC

Cognition • Singapore

On-site
SGD 60,000 - 100,000