Staff Software Engineer - Web-Scale Crawling & Ingestion

Nebius Group; City of Amsterdam (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Competitive compensation

Career growth and learning

Flexibility and ownership

Collaborative and innovative culture

Impactful AI projects

International environment

Job summary

Nebius is building a cutting-edge AI cloud platform and is hiring a Staff Software Engineer to lead content acquisition and crawling infrastructure for a scale-out search engine designed for agentic AI consumption.

You will design and operate distributed crawling, ingestion, and scheduling systems handling billions of URLs, with a focus on freshness, coverage, and resource efficiency. Join a fast-moving, globally distributed team at Nebius.

Qualifications

• 8+ years of experience building backend or distributed systems.
• Strong Go or C++ expertise.
• Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines).
• Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction.

Responsibilities

• Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
• Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
• Ensure reliable operation of crawling infrastructure under high-throughput conditions.
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
• Monitor resource usage, bandwidth consumption, and infrastructure cost.
• Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements.
• Enable safe experimentation with crawling strategies and content acquisition policies.

Skills

Go

C++

Distributed systems

Backend development

Tools

Kafka

Pulsar

NATS

RabbitMQ

Spark

Flink

Beam

MapReduce

Job description

Nebius is building a cutting-edge AI cloud platform and is hiring a Staff Software Engineer to lead content acquisition and crawling infrastructure for a scale-out search engine designed for agentic AI consumption.

You will design and operate distributed crawling, ingestion, and scheduling systems handling billions of URLs, with a focus on freshness, coverage, and resource efficiency. Join a fast-moving, globally distributed team at Nebius.