Staff / Senior Software Engineer (Agentic Search) - Index

nebius; Stati Uniti; Remoto; USD 180.000 - 260.000; Tempo pieno

Descrizione del lavoro

nebius is seeking a senior or staff-level software engineer to own the indexing and data processing layer of a purpose-built AI-focused search platform. You will design and operate high-throughput pipelines, with a focus on freshness, completeness, and efficient querying across tens of gigabytes per second.

The role emphasizes systems design, distributed data processing, and collaboration with ML teams to ensure retrieved data meets ranking requirements.

Competenze

• Five+ years of experience building production backend or data infrastructure systems.
• Strong proficiency in Go; working knowledge of C++ or Rust is a plus.
• Experience with large-scale data processing at multi-GB/sec throughput and petabyte-scale datasets.
• Background building or operating databases, storage systems, data planes, or indexing pipelines.

Mansioni

• Design, build, and operate large-scale indexing systems and data pipelines at the core of the search infrastructure.
• Develop and tune indexing strategies balancing performance, freshness, and resource efficiency.
• Define and implement observability primitives and monitor throughput, resource usage, and cost over time.
• Build well-tested components with clear contracts and safe experimentation through controlled rollouts.

Conoscenze

Go

C++

Rust

Distributed systems

Data processing

Strumenti

Spark

Flink

Beam

MapReduce

Descrizione del lavoro

Role overview

A senior or staff-level software engineer is sought to own the indexing and data processing layer of a search engine purpose-built for AI agents rather than human users. The platform exposes programmatic, low-latency, observable search APIs that other AI systems use to retrieve and reason over real-world information. The role is heavily systems-oriented, focused on offline and nearline pipelines that must move tens of gigabytes per second while keeping data fresh, complete, and efficiently queryable.

Responsibilities

• Design, build, and operate large-scale indexing systems and data pipelines at the core of the search infrastructure.
• Develop and tune indexing strategies that balance performance, freshness, and resource efficiency, including storage formats, compaction strategies, and update mechanisms.
• Define and implement observability primitives such as structured logs, metrics, and data-quality signals, and monitor throughput, resource usage, and cost over time.
• Build well-tested components with clear contracts, enabling safe experimentation through controlled rollouts and clearly defined quality signals.
• Collaborate with runtime and machine-learning teams to ensure indexed outputs meet retrieval and ranking requirements.
• Drive reliability work, incident response, and ongoing optimisations for pipelines operating under sustained high throughput.

Requirements

• Five or more years of experience building production backend or data infrastructure systems.
• Strong proficiency in Go, with working knowledge of C++ or Rust considered a plus.
• Hands-on experience with large-scale data processing at multi-gigabyte-per-second throughput and petabyte-scale datasets.
• Background building or operating databases, storage systems, data planes, or indexing pipelines.
• Deep understanding of distributed systems, including fault tolerance, consistency, and scalability.
• Comfort running production systems and handling operational incidents, with a systems-thinking mindset for end-to-end data flows.

Nice to have

• Experience with distributed data processing frameworks such as Spark, Flink, MapReduce, or Beam.
• Familiarity with content systems including scraping, proxying, or anti-bot infrastructure, or backgrounds in ad tech, social platforms, or other large-scale content systems.
• Knowledge of DBMS internals and cloud infrastructure, open-source contributions, competitive programming or CTF experience, advanced technical programmes, conference talks, or technical publications.

Benefits and work setup

• Competitive compensation with career growth and learning opportunities.
• Flexible, high-ownership culture within an international, collaborative engineering environment, focused on impactful AI infrastructure projects.