Senior Software Engineer — Large-Scale Data Pipelines (Go)

Nebius; Greater London; On-site; GBP 110,000 - 165,000; Full time

Benefits offered by this job

Competitive pay

Career growth

Flexibility and ownership

Collaborative culture

Impactful AI projects

International teams

Job summary

Nebius is building a cutting-edge agent‑native search platform in the cloud. We are seeking a Senior Software Engineer to join the indexing and data processing layer, handling massive data ingestion and organization for fast, scalable retrieval.

You will design and implement large‑scale indexing systems, optimise data pipelines, and collaborate with ML teams to meet retrieval requirements. Expect high throughput work, observable systems, and opportunities to shape the future of AI infrastructure

Qualifications

• 5+ years of experience building production backend or data infrastructure systems.
• Strong Go experience (C++/Rust is a plus).
• Experience with large-scale data processing systems (10+ GiB/sec throughput, petabyte-scale datasets).
• Experience building or operating databases, storage systems, data planes, or indexing pipelines.
• Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
• Experience running production systems and handling operational incidents.
• Systems-thinking mindset and ability to reason about end-to-end data flows.

Responsibilities

• Design, implement, and operate large-scale indexing systems and data pipelines.
• Develop and optimise indexing strategies balancing performance, freshness, and resource efficiency.
• Work on storage formats, compaction, and updates to keep data accessible.
• Ensure reliability and predictability of pipelines under high-throughput conditions.
• Build well-tested components with clear responsibilities and interaction contracts.
• Define and implement observability primitives, including structured logs, metrics, and data quality signals.
• Monitor throughput, resource usage, and cost, and drive optimisations when needed.
• Collaborate with runtime and ML teams to ensure indexing outputs meet retrieval requirements.
• Enable safe experimentation on indexing strategies and data processing logic through controlled rollouts.

Skills

5+ years exp

Go experience

Large-scale data

Databases & storage

Distributed systems

Production operations

Systems thinking

Tools

Spark

Flink

MapReduce

Beam

Job description

Nebius is building a cutting-edge agent‑native search platform in the cloud. We are seeking a Senior Software Engineer to join the indexing and data processing layer, handling massive data ingestion and organization for fast, scalable retrieval.

You will design and implement large‑scale indexing systems, optimise data pipelines, and collaborate with ML teams to meet retrieval requirements. Expect high throughput work, observable systems, and opportunities to shape the future of AI infrastructure