Senior Software Engineer — Distributed Web Crawling & Ingestion

Nebius Group; Zürich; Vor Ort; USD 140.000 - 210.000; Vollzeit

Benefits dieser Stelle

Competitive pay

Career growth

Flexibility

Collaborative culture

Impactful AI projects

International teams

Zusammenfassung

Nebius Group is building a next-generation AI cloud platform and an agent-native search engine. We are hiring a Senior Software Engineer to design and operate content acquisition and crawling infrastructure that scales to internet-level content, balancing freshness, coverage, and efficiency.

You will work across distributed systems, ingestion pipelines, and observability while collaborating with ML and indexing teams to ensure high-quality data for retrieval and ranking.

Qualifikationen

• 5+ years of experience building backend or distributed systems.
• Strong Go or C++ expertise.
• Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines).

Aufgaben

• Design, implement, and operate web-scale crawling systems for acquiring content from the internet.
• Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
• Ensure reliable operation of crawling infrastructure under high-throughput conditions.
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
• Monitor resource usage, bandwidth consumption, and infrastructure cost.
• Collaborate with indexing and ML teams to ensure retrieved content meets retrieval and ranking needs.
• Enable safe experimentation with crawling strategies and content acquisition policies.

Kenntnisse

Go

C++

Distributed systems

Web protocols

Production systems

Tools

Kafka

Pulsar

NATS

RabbitMQ

Jobbeschreibung

Nebius Group is building a next-generation AI cloud platform and an agent-native search engine. We are hiring a Senior Software Engineer to design and operate content acquisition and crawling infrastructure that scales to internet-level content, balancing freshness, coverage, and efficiency.

You will work across distributed systems, ingestion pipelines, and observability while collaborating with ML and indexing teams to ensure high-quality data for retrieval and ranking.