Staff Software Engineer: Agentic AI Search & Crawling

Nebius Group; Greater London; Presencial; GBP 120.000 - 190.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Competitive pay

Career growth

Flexibility

Collaborative culture

Impactful AI projects

International environment

Descripción de la vacante

Nebius is building an agent-native search platform and a content acquisition pipeline designed for AI systems. We seek a Staff Software Engineer to design and operate scalable crawling, ingestion, and processing infrastructure for billions of URLs and real-world data.

You will build distributed, high-throughput systems, ensure reliability, and collaborate with ML and indexing teams to optimize retrieval, ranking, and freshness of content at scale.

Formación

• 8+ years of experience building backend or distributed systems.
• Strong Go or C++ expertise.
• Experience with large-scale distributed systems and high-throughput pipelines.
• Understanding of web protocols (HTTP, DNS, TLS) and content processing.
• Experience operating production systems and debugging distributed failures.
• Strong knowledge of scalability, fault tolerance, and resource management.

Responsabilidades

• Design, implement, and operate web-scale crawling and data ingestion systems for content acquisition.
• Build ingestion workflows for diverse data sources and partner integrations.
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
• Create URL discovery, deduplication, content extraction, and crawl orchestration systems.
• Ensure reliable operation of crawling infrastructure under high throughput.
• Define observability and quality metrics for crawl coverage and content quality.
• Monitor resource usage, bandwidth, and infra costs.
• Collaborate with indexing and ML teams to meet retrieval and ranking needs.
• Enable safe experimentation with crawling policies and policies.

Conocimientos

Go

C++

Distributed systems

Web protocols

Herramientas

Kafka

Pulsar

NATS

RabbitMQ

Spark

Flink

Descripción del empleo

Nebius is building an agent-native search platform and a content acquisition pipeline designed for AI systems. We seek a Staff Software Engineer to design and operate scalable crawling, ingestion, and processing infrastructure for billions of URLs and real-world data.

You will build distributed, high-throughput systems, ensure reliability, and collaborate with ML and indexing teams to optimize retrieval, ranking, and freshness of content at scale.