Staff Software Engineer - Scalable Data Pipelines

Anyscale; San Francisco, Northern (CA, KY); Hybrid; USD 170,000 - 260,000; Full time

Job summary

Anyscale is seeking exceptional engineers to design, build, and scale Ray Data, a Python-native data processing engine powering modern AI workloads. You will work on core systems, optimize distributed execution, and collaborate with customers to solve hard scalability challenges.

The role requires 6+ years of production-grade software experience with Python, deep knowledge of distributed systems, and a track record of architectural decision making in data or compute engines.

Qualifications

• 6+ years of production-grade software or infrastructure experience.
• Strong Python engineering experience.
• Experience owning architectural decisions in distributed data or compute engines.

Responsibilities

• Design, build, and optimize core Ray Data systems for performance and scalability.
• Work on distributed execution across data pipelines.
• Develop data loading and processing for training and inference workloads.
• Address fault tolerance, partitioning, and resource management.

Skills

Python

Distributed systems

Performance optimization

Scheduling

Tools

Ray

Linux

Docker

Job description

Anyscale is seeking exceptional engineers to design, build, and scale Ray Data, a Python-native data processing engine powering modern AI workloads. You will work on core systems, optimize distributed execution, and collaborate with customers to solve hard scalability challenges.

The role requires 6+ years of production-grade software experience with Python, deep knowledge of distributed systems, and a track record of architectural decision making in data or compute engines.