Software engineer in distributed storage

Nebius; United States; Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Hybrid work arrangement

Remote within EU

Job summary

Nebius Group ищет инженера C/C++ для работы над YDB — высокопроизводительной системе для облачной инфраструктуры искусственного интеллекта. Ваша задача — развивать архитектуру, оптимизировать хранение данных и сеть, работать с современными технологиями и обеспечивать надёжность в крупных кластерах.

Участие в проектах с распределёнными системами, установка и настройка инструментов профилирования, участие в развитии инфраструктуры и процессов развёртывания — всё это ожидается на позиции.

Qualifications

• 5+ лет профессионального опыта разработки высоконагруженных систем на C/C++
• Опыт в распределённых системах хранения данных
• Знание CPU кэшей, NUMA и атомарности
• Понимание инструментов профилирования: perf, VTune, bpftrace, gdb
• Понимание Linux kernel, SPDK/DPDK, libaio и io_uring
• Опыт работы с Kubernetes и Grafana
• Опыт реагирования на инциденты в продуктивной среде
• Знание IP/TCP/UDP; InfiniBand/ROCE/RDMA желателено

Responsibilities

• Эволюция архитектуры YDB для нового аппаратного обеспечения
• Улучшение хранения данных и обработка ошибок с учетом новых дисков
• Оптимизация сетевых обходов и производительности CPU/памяти
• Улучшение устойчивости к сбоям и ускорение восстановления после сбоев
• Обеспечение интерфейсов и автоматизации развёртывания множества кластеров

Skills

C/C++ high-load systems

Distributed storage

Performance optimization

Linux system programming

Networking basics

Tools

perf

VTune

bpftrace

gdb

SPDK/DPDK

io_uring

Kubernetes

Grafana

InfiniBand/ROCE

Job description

Описание

Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure. The YDB Development Team at Nebius is hiring mid- and senior-level software developers with C++ expertise.

Задачи

Evolve YDB’s architecture for new hardware, redesigning core components when incremental optimization is insufficient: QLC NVMe drives: increase storage density while controlling RAM/CPU consumption and write amplification, Dual 800 Gb/s network adapters: reduce CPU overhead and memory bandwidth consumption to utilize the available NIC bandwidth, NVIDIA GB300-based systems: ensure efficient operation on Arm-based hosts and offload storage and networking processing to BlueField DPUs Support high-capacity SMR HDDs by designing data placement, sequential writes, garbage collection and recovery around zoned-write constraints Develop wide erasure-coding schemes to reduce space aplimfication while maintaining durability and availability Reduce recovery and rebalancing times after device outages while limiting the impact on foreground workloads Prevent, detect and recover from silent data corruption across memory, networking and storage Provide the system capabilities and interfaces required by fleet-wide automation so that hundreds of production clusters can be deployed, configured, upgraded, diagnosed and operated consistently

Требования

5+ years of professional experience developing high-load systems in C or C++. Candidates with 3+ years will also be considered if their experience includes relevant work on distributed storage systems Experience developing high-performance, low-latency software components Understanding of core dumps, flame graphs and sanitizer-enabled builds Understanding of CPU caches, atomic operations and NUMA architectures Будет плюсом: Experience designing and developing production-grade distributed storage systems and/or databases; Experience with on-disk data structures such as LSM trees and B+ trees, and storage algorithms such as erasure coding and checksumming; Understanding of NVMe, conventional HDD and SMR HDD internals; Experience with debugging and profiling tools such as perf, VTune, bpftrace and gdb; Familiarity with the Linux kernel, SPDK/DPDK, libaio and io_uring; Knowledge of IP, TCP, UDP and DNS, and ideally InfiniBand or RoCE/RDMA; Experience with Kubernetes and Grafana; Experience responding to production incidents; Knowledge of availability and durability calculations

Условия

For Senior candidates, we offer hybrid working or the flexibility to work remotely within EU. Competitive compensation Career growth and learning opportunities Flexibility and ownership Collaborative and innovative culture Opportunity to work on impactful AI projects International environment and talented teams