Member of Technical Staff (Web Access Engineer, Crawler)

Perplexity

London

On-site

CAD 110,000 - 160,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Perplexity is seeking a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems powering AI-powered search. You will diagnose network and rendering issues, implementing production fixes that improve content quality at scale.

You will investigate DNS, TLS, HTTP, caching, redirects, and client-side rendering, ensuring access policies are respected while advancing crawling performance and reliability.

Qualifications

  • 5+ years in software engineering for web infrastructure, browsers, CDNs or crawling.
  • Strong programming skills in Go, Python, Rust or C++.
  • Experience debugging web rendering and network stack.

Responsibilities

  • Own fetching and rendering improvements end to end, from investigation to production rollout.
  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering.
  • Improve page loading, content capture, and completion detection for dynamic pages.
  • Ensure crawler respects site access restrictions and rate limits.
  • Build tools to reproduce failures and create regression tests.
  • Define metrics for fetching reliability and content completeness.

Skills

Web infrastructure
Go
Python
Rust
C++
Browser debugging

Tools

Chrome DevTools
CDP
HAR files
curl

Job description

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.

Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.

Responsibilities
  • Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
  • Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
  • Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
  • Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
  • Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.
Qualifications
  • 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.
  • Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.
  • Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.
  • Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
  • Comfort working independently when the path forward is not fully defined.
Nice to have
  • Experience developing CDN, caching, or web performance systems.
  • Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.
  • Contributions to a browser engine or JavaScript runtime.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Web Crawler & Fetch/Render Reliability Engineer
Web Crawler & Fetch/Render Reliability Engineer

Perplexity • London

On-site
CAD 110,000 - 160,000
Member of Technical Staff (Site Reliability Engineer, Crawler)
Member of Technical Staff (Site Reliability Engineer, Crawler)

Perplexity • London

On-site
CAD 120,000 - 160,000
Software Developer (Content Integration & Data Pipelines)
Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions LLC • Ottawa

Remote
CAD 75,000 - 110,000
RRSP matching
Life Insurance
Employee Assistance Program (EAP)
+1
Junior Software Developer (Content Integration & Data Pipelines)
Junior Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions LLC • Ottawa

Remote
CAD 65,000 - 75,000
Fully remote work environment
RRSP matching
Life Insurance
+3
Senior Staff Software Engineer (Browser, New Products)
Senior Staff Software Engineer (Browser, New Products)

Mozilla • Ottawa

On-site
CAD 120,000 - 160,000
Health and wellness
Mental health
Time away
+7
Junior Software Developer (Content Integration & Data Pipelines)
Junior Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions • Ottawa

Remote
CAD 65,000 - 75,000
RRSP matching
Life Insurance
EAP
Junior Software Developer (Content Integration & Data Pipelines)
Junior Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions • Ottawa

Remote
CAD 65,000 - 75,000
Remote-friendly
RRSP matching
Life Insurance
+3
Search Engineer
Search Engineer

Firecrawl • Toronto

On-site
CAD 217,000 - 233,000
Competitive equity
15 days PTO
Parental leave
+4
Software Developer (Content Integration & Data Pipelines)
Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions • Ottawa

Remote
CAD 75,000 - 110,000
Fully remote work
RRSP matching
Life Insurance
+4
Software Developer (Content Integration & Data Pipelines)
Software Developer (Content Integration & Data Pipelines)

Agility PR Solutions • Ottawa

Remote
CAD 75,000 - 110,000
Fully remote work environment
RRSP matching
Life Insurance
+2