Member of Technical Staff (Index Engineer, Search Crawler)

Perplexity

Greater London

On-site

GBP 90,000 - 140,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Perplexity seeks an engineer to work on the indexing and data processing layer of its Search Crawler. You will design and implement large-scale indexing systems and data pipelines, ingesting crawled content into efficient structures and ensuring freshness and accessibility for downstream retrieval.

You should have 5+ years building production backend or data infrastructure, with strong Go, C++, or Rust experience, and familiarity with high‑throughput data and distributed systems.

Qualifications

  • 5+ years of experience building production backend or data infrastructure systems.
  • Strong programming experience in Go, C++, or Rust.
  • Experience with high-throughput data processing systems and very large datasets.
  • Experience building or operating databases, storage systems, data planes, or indexing pipelines.
  • Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
  • Experience running production systems and handling operational incidents.
  • A systems-thinking mindset and the ability to reason about end-to-end data flows.

Responsibilities

  • Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
  • Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
  • Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
  • Keep pipelines reliable and predictable under high-throughput conditions.
  • Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
  • Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
  • Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
  • Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
  • Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.

Skills

Backend data infra
Go
C++
Rust
High-throughput data processing
Distributed systems
Operational incident handling

Tools

Databases
Storage systems
Indexing pipelines

Job description

AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.

Responsibilities
  • Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
  • Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
  • Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
  • Keep pipelines reliable and predictable under high-throughput conditions.
  • Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
  • Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
  • Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
  • Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
  • Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.
Qualifications
  • 5+ years of experience building production backend or data infrastructure systems.
  • Strong programming experience in Go, C++, or Rust.
  • Experience with high-throughput data processing systems and very large datasets.
  • Experience building or operating databases, storage systems, data planes, or indexing pipelines.
  • Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
  • Experience running production systems and handling operational incidents.
  • A systems-thinking mindset and the ability to reason about end-to-end data flows.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Engineer: Large-Scale Data Indexing & Pipelines
Staff Engineer: Large-Scale Data Indexing & Pipelines

Perplexity • Greater London

On-site
GBP 90,000 - 140,000
Member of Technical Staff (Search Engine Developer, Search Core)
Member of Technical Staff (Search Engine Developer, Search Core)

Perplexity • Greater London

On-site
GBP 90,000 - 130,000
Member of Technical Staff (Backend/Infrastructure Engineer, Search)
Member of Technical Staff (Backend/Infrastructure Engineer, Search)

Perplexity • Greater London

On-site
GBP 90,000 - 120,000
Senior Systems Engineer, Search Core & Indexing
Senior Systems Engineer, Search Core & Indexing

Perplexity • Greater London

On-site
GBP 90,000 - 130,000
Member of Technical Staff (Software Engineer, Fraud Prevention)
Member of Technical Staff (Software Engineer, Fraud Prevention)

Perplexity AI • Greater London

On-site
GBP 85,000 - 140,000
Member of Technical Staff (Search Crawler Analyst)
Member of Technical Staff (Search Crawler Analyst)

Perplexity • Greater London

On-site
GBP 65,000 - 90,000
Member of Technical Staff (Software Engineer, Infrastructure & Platform)
Member of Technical Staff (Software Engineer, Infrastructure & Platform)

Perplexity • Greater London

On-site
GBP 90,000 - 130,000
Member of Technical Staff (Site Reliability Engineer, Crawler)
Member of Technical Staff (Site Reliability Engineer, Crawler)

Academic Key • Greater London

On-site
GBP 90,000 - 150,000
1d Perplexity Member of Technical Staff (Site Reliability Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Site Reliability Engineer, Crawler) London
1d Perplexity Member of Technical Staff (Site Reliability Engineer, Crawler) London Perplexity 1d Member of Technical Staff (Site Reliability Engineer, Crawler) London

Applied Methods Ltd • Greater London

On-site
GBP 110,000 - 140,000
Backend Engineer - Web Crawling & Search Platform
Backend Engineer - Web Crawling & Search Platform

Perplexity • Greater London

On-site
GBP 90,000 - 120,000