Senior Software Engineer: Web-Scale Crawling & Ingestion

Nebius; Zürich; Vor Ort; CHF 180.000 - 240.000; Vollzeit

Benefits dieser Stelle

Competitive compensation

Career growth and learning

Flexibility and ownership

Collaborative and innovative culture

Impactful AI projects

Zusammenfassung

Nebius, a leading cloud AI infrastructure company, is seeking a Senior Software Engineer to own the content acquisition and web-scale crawling infrastructure for an agent-native search platform. You will design and operate distributed crawling, ingestion, and scheduling systems at internet scale, ensuring fresh, high-quality content for indexing and retrieval.

You will collaborate with indexing and ML teams, optimize resource usage, and define observability metrics.

Qualifikationen

• 5+ years of experience building backend or distributed systems.
• Strong Go or C++ expertise.
• Experience with large-scale distributed systems (10k+ RPS, billions of URLs).
• Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction.
• Experience operating production systems and debugging failures in distributed environments.
• Strong understanding of scalability, fault tolerance, and resource management.

Aufgaben

• Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
• Build ingestion workflows for internal and external data sources, including crawlers and feeds.
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
• Ensure reliable operation of crawling infrastructure under high-throughput conditions.
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
• Monitor resource usage, bandwidth, and infrastructure cost.
• Collaborate with indexing and ML teams to ensure content meets retrieval and ranking requirements.
• Enable safe experimentation with crawling strategies and content acquisition policies.

Kenntnisse

Backend systems

Distributed systems

Go

C++

HTTP/DNS/TLS

Production systems

Tools

Kafka

Pulsar

NATS

RabbitMQ

Jobbeschreibung

Nebius, a leading cloud AI infrastructure company, is seeking a Senior Software Engineer to own the content acquisition and web-scale crawling infrastructure for an agent-native search platform. You will design and operate distributed crawling, ingestion, and scheduling systems at internet scale, ensuring fresh, high-quality content for indexing and retrieval.

You will collaborate with indexing and ML teams, optimize resource usage, and define observability metrics.