Petabyte-Scale Data Infrastructure Engineer

Cohere; Greater London; Hybrid; GBP 120,000 - 170,000; Full time

Benefits offered by this job

Lunch stipend $75/£75 weekly

Health and dental benefits

RRSP matching / Pension

Parental leave top-up

Education & learning stipend

6 weeks paid vacation

Home office stipend

Job summary

Cohere is a leading security-first enterprise AI company building cutting-edge foundation models and end-to-end products. We are looking for a data infrastructure engineer to design and operate the storage layer powering training runs.

You will deploy this system across Kubernetes clusters at petabyte scale, collaborate with researchers to optimize dataflow, and address cross-region data movement challenges to maximize throughput and minimize downtime.

Qualifications

• Strong storage fundamentals, including replication, consistency, caching, and data lifecycle management.
• Strong coding ability. We work in Python and Go; experience in either is enough, but you should be willing to pick up the other
• Experience running stateful systems on Kubernetes, including Persistent Volumes, CSI drivers, and StatefulSets.
• Hands-on experience with cloud object storage such as S3 as well as POSIX-style filesystems.

Responsibilities

• Design, build, and operate the distributed storage system that feeds model training and evaluation.
• Run this system multiple on Kubernetes clusters at petabyte scale.
• Work with researchers and training-infra teams on how jobs actually read and write data, and turn that into throughput, latency, and durability requirements
• Work through the networking, I/O, and consistency problems of moving large datasets and checkpoints across regions and backends, with GPU idle time and time-to-insight as the measures of success

Skills

Storage fundamentals

Go or Python

Kubernetes experience

Cloud storage (S3)

Distributed systems

Tools

Kubernetes

S3

POSIX filesystems

Job description

Cohere is a leading security-first enterprise AI company building cutting-edge foundation models and end-to-end products. We are looking for a data infrastructure engineer to design and operate the storage layer powering training runs.

You will deploy this system across Kubernetes clusters at petabyte scale, collaborate with researchers to optimize dataflow, and address cross-region data movement challenges to maximize throughput and minimize downtime.