1d Perplexity Member of Technical Staff (Web Access Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Web Access Engineer, Crawler) London

Applied Methods Ltd

Greater London

Remote

GBP 70,000 - 100,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You will diagnose network errors, page-loading issues, and incomplete rendering, working across DNS, TLS, HTTP, caching, and client-side execution.

You will own end-to-end improvements, create regression tests, and define metrics for fetching reliability and content completeness at scale in London with remote options.

Qualifications

  • 5+ years of software engineering in web infrastructure or crawling.
  • Experience with HTTP/1.1, HTTP/2, HTTP/3 and TLS.
  • Proficiency in Go, Python, Rust or C++.
  • Familiarity with CDNs, caching and rate limits.

Responsibilities

  • Own fetching and rendering improvements end to end from investigation to production release.
  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects and client-side rendering.
  • Improve page loading, completion detection and content capture for dynamic pages.
  • Define metrics for fetching reliability and content completeness; build regression tests.

Skills

Go
Python
Rust
C++
Chrome DevTools
HAR files
CDP
curl

Tools

Chrome DevTools
HAR files
CDP
curl

Job description

Perplexity# Member of Technical Staff (Web Access Engineer, Crawler)SearchLondonRemoteFull-TimePosted yesterday## About the rolePerplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.**Responsibilities*** Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.* Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.* Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.* Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.* Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.* Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.**Qualifications*** 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.* Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.* Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.* Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.* Comfort working independently when the path forward is not fully defined.**Nice to have*** Experience developing CDN, caching, or web performance systems.* Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.* Contributions to a browser engine or JavaScript runtime.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff (Software Engineer, Fraud Prevention)
Member of Technical Staff (Software Engineer, Fraud Prevention)

Perplexity AI • Greater London

On-site
GBP 85,000 - 140,000
1d Perplexity Member of Technical Staff (Site Reliability Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Site Reliability Engineer, Crawler) London
1d Perplexity Member of Technical Staff (Site Reliability Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Site Reliability Engineer, Crawler) London

Applied Methods Ltd • Greater London

On-site
GBP 110,000 - 140,000
Remote Web Access Engineer (Crawler) - Scalable Fetching
Remote Web Access Engineer (Crawler) - Scalable Fetching

Applied Methods Ltd • Greater London

Remote
GBP 70,000 - 100,000
Member of Technical Staff (Site Reliability Engineer, Crawler)
Member of Technical Staff (Site Reliability Engineer, Crawler)

Academic Key • Greater London

On-site
GBP 90,000 - 150,000
Senior Web Access Engineer - Crawler Infrastructure
Senior Web Access Engineer - Crawler Infrastructure

Perplexity AI • Greater London

On-site
GBP 85,000 - 140,000
Member of Technical Staff (Index Engineer, Search Crawler)
Member of Technical Staff (Index Engineer, Search Crawler)

Perplexity • Greater London

On-site
GBP 90,000 - 140,000
Member of Technical Staff (Site Reliability Engineer, Crawler)
Member of Technical Staff (Site Reliability Engineer, Crawler)

Perplexity • Greater London

On-site
GBP 90,000 - 125,000
Member of Technical Staff (Search Crawler Analyst)
Member of Technical Staff (Search Crawler Analyst)

Perplexity • Greater London

On-site
GBP 65,000 - 90,000
Staff Engineer: Large-Scale Data Indexing & Pipelines
Staff Engineer: Large-Scale Data Indexing & Pipelines

Perplexity • Greater London

On-site
GBP 90,000 - 140,000
Member of Technical Staff (Backend Software Engineer, Search Crawler)
Member of Technical Staff (Backend Software Engineer, Search Crawler)

Perplexity • Greater London

On-site
GBP 90,000 - 120,000