Senior Web Access Engineer - Crawler Infrastructure

Perplexity AI

Greater London

On-site

GBP 85,000 - 140,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Perplexity AI is seeking a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You will diagnose network errors, page-loading issues, and incomplete rendering, tracing problems from network protocols through client-side execution to production fixes that improve content quality at scale.

You will own end-to-end improvements, investigate failures across DNS, TLS, HTTP, caching, and redirects, and ensure respectful access with proper

Qualifications

  • 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling.
  • Strong programming skills in Go, Python, Rust, or C++.
  • Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including TLS, DNS, cookies, and sessions.
  • Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; debugging JavaScript with Chrome DevTools, CDP, HAR files, curl, and packet captures.
  • Comfort working independently when the path forward is not fully defined.

Responsibilities

  • Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
  • Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
  • Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
  • Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
  • Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.

Skills

years of software engineering
Go
Python
Rust
C++
HTTP/1.1
HTTP/2
HTTP/3
TLS
DNS
Cookies & Sessions
Chrome DevTools

Tools

HAR files
curl
CDP
packet captures
CDN caching
SRE collaboration

Job description

Perplexity AI is seeking a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You will diagnose network errors, page-loading issues, and incomplete rendering, tracing problems from network protocols through client-side execution to production fixes that improve content quality at scale.

You will own end-to-end improvements, investigate failures across DNS, TLS, HTTP, caching, and redirects, and ensure respectful access with proper

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Web Access Engineer (Crawler) - Scalable Fetching
Remote Web Access Engineer (Crawler) - Scalable Fetching

Applied Methods Ltd • Greater London

Remote
GBP 70,000 - 100,000
Member of Technical Staff (Software Engineer, Fraud Prevention)
Member of Technical Staff (Software Engineer, Fraud Prevention)

Perplexity AI • Greater London

On-site
GBP 85,000 - 140,000
1d Perplexity Member of Technical Staff (Web Access Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Web Access Engineer, Crawler) London
1d Perplexity Member of Technical Staff (Web Access Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Web Access Engineer, Crawler) London

Applied Methods Ltd • Greater London

Remote
GBP 70,000 - 100,000
Backend Engineer - Web Crawling & Search Platform
Backend Engineer - Web Crawling & Search Platform

Perplexity • Greater London

On-site
GBP 90,000 - 120,000
Member of Technical Staff (Backend Software Engineer, Search Crawler)
Member of Technical Staff (Backend Software Engineer, Search Crawler)

Perplexity • Greater London

On-site
GBP 90,000 - 120,000
Remote: Research Engineer, Large-Scale Web Crawling
Remote: Research Engineer, Large-Scale Web Crawling

ElevenLabs • Greater London

Hybrid
GBP 80,000 - 120,000
Innovative culture
Growth paths
Learning & development
+3
Site Reliability Engineer — Crawler Platforms & Reliability
Site Reliability Engineer — Crawler Platforms & Reliability

Perplexity • Greater London

On-site
GBP 90,000 - 125,000
Staff Engineer: Large-Scale Data Indexing & Pipelines
Staff Engineer: Large-Scale Data Indexing & Pipelines

Perplexity • Greater London

On-site
GBP 90,000 - 140,000
Staff Software Engineer, Web-Scale Crawling & Ingestion
Staff Software Engineer, Web-Scale Crawling & Ingestion

FDE Times • Greater London

Hybrid
GBP 120,000 - 190,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+3
Member of Technical Staff (Index Engineer, Search Crawler)
Member of Technical Staff (Index Engineer, Search Crawler)

Perplexity • Greater London

On-site
GBP 90,000 - 140,000