Staff Backend Engineer – Agentic Web Crawling (Flexible)

FDE Times

Amsterdam

Hybrid

EUR 110,000 - 170,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Competitive compensation
Career growth and learning potential
Flexibility and ownership
Collaborative culture
Impactful AI projects
International environment

Job summary

Nebius is building a cutting-edge, agent-native search platform for AI systems, seeking a Staff Software Engineer to help design and operate content acquisition and crawling infrastructure. You will tackle distributed crawling, scheduling, ingestion, and high-throughput data flows to power indexing and retrieval for billions of URLs.

Strong Go or C++ and production-grade distributed systems experience are essential.

Qualifications

  • 8+ years of experience building backend or distributed systems.
  • Strong Go or C++ expertise.
  • Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines).
  • Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction.
  • Experience operating production systems and debugging failures in distributed environments.
  • Strong understanding of scalability, fault tolerance, and resource management.

Responsibilities

  • Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
  • Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
  • Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
  • Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
  • Ensure reliable operation of crawling infrastructure under high-throughput conditions.
  • Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
  • Monitor resource usage, bandwidth consumption, and infrastructure cost.
  • Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements.
  • Enable safe experimentation with crawling strategies and content acquisition policies.

Skills

Go
C++
Backend systems
Distributed systems

Tools

Kafka
Pulsar
NATS
RabbitMQ

Job description

Nebius is building a cutting-edge, agent-native search platform for AI systems, seeking a Staff Software Engineer to help design and operate content acquisition and crawling infrastructure. You will tackle distributed crawling, scheduling, ingestion, and high-throughput data flows to power indexing and retrieval for billions of URLs.

Strong Go or C++ and production-grade distributed systems experience are essential.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Engineer — Web-Scale Crawling & Ingestion
Staff Engineer — Web-Scale Crawling & Ingestion

Nebius Group • Amsterdam

Remote
EUR 120,000 - 180,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+3
Staff Software Engineer: Web-Scale Crawling & Ingestion
Staff Software Engineer: Web-Scale Crawling & Ingestion

Nebius Group • Amsterdam

On-site
EUR 150,000 - 190,000
Competitive pay
Career growth
Flexibility
+3
Staff Software Engineer (Agentic Search) - Crawler (opens in a new tab) Amsterdam
Staff Software Engineer (Agentic Search) - Crawler (opens in a new tab) Amsterdam

Nebius Group • Amsterdam

Remote
EUR 120,000 - 180,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+3
Staff Software Engineer (Agentic Search) - Crawler
Staff Software Engineer (Agentic Search) - Crawler

FDE Times • Amsterdam

Hybrid
EUR 110,000 - 170,000
Competitive compensation
Career growth and learning potential
Flexibility and ownership
+3
Staff Software Engineer (Agentic Search) - Crawler
Staff Software Engineer (Agentic Search) - Crawler

Nebius Group • Amsterdam

On-site
EUR 150,000 - 190,000
Competitive pay
Career growth
Flexibility
+3
Staff Software Engineer - AI Search & Data Pipelines
Staff Software Engineer - AI Search & Data Pipelines

Nebius Group • Amsterdam

Remote
EUR 120,000 - 180,000
Competitive compensation
Career growth
Flexible work
+3
Staff Software Engineer: Real-Time Agentic Search Backend
Staff Software Engineer: Real-Time Agentic Search Backend

Nebius Group • Amsterdam

Remote
EUR 120,000 - 160,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+2
Staff Engineer - Agentic AI Search Runtime
Staff Engineer - Agentic AI Search Runtime

Rustify • Amsterdam

Remote
EUR 120,000 - 170,000
Senior Web Access Engineer for AI-Scale Content Fetching
Senior Web Access Engineer for AI-Scale Content Fetching

FDE Times • Amsterdam

On-site
EUR 90,000 - 130,000
Competitive compensation
Career growth opportunities
Flexibility and ownership
+3
Staff Applied AI Researcher: Agentic Search & Retrieval
Staff Applied AI Researcher: Agentic Search & Retrieval

Nebius Group • Amsterdam

Hybrid
EUR 140,000 - 190,000
Competitive compensation
Career growth and learning
Flexibility and ownership
+3