Software Engineer, Ray Data & AI Pipelines

Anyscale, Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Anyscale Inc. seeks engineers for the Ray Data Team to advance Python-native data processing for scalable ML workloads. You will help scale Ray from laptop to cluster, collaborating with AI-native customers to push performance and reliability in production environments.

Responsibilities focus on improving Ray Data, scaling across pipeline stages, and building loading solutions for training workloads, with a strong emphasis on stability and fault tolerance at scale.

Qualifications

• At least 3-4 years of relevant work experience.
• Solid background in building scalable and fault-tolerant distributed systems.
• Experience with data processing, database internals.
• Passionate about large scale systems and performance for AI.

Responsibilities

• Improve the performance of Ray Data and multi-modal batch inference use cases.
• Ensure efficient scaling across different stages of the Data pipeline in a heterogeneous environment.
• Building data loading solutions for production training workloads.
• Focus on stability and fault tolerance at high scale
• Working with customers and new age AI native companies in scaling their AI workloads.

Skills

Distributed systems

Data processing

Performance optimization

Job description

Anyscale Inc. seeks engineers for the Ray Data Team to advance Python-native data processing for scalable ML workloads. You will help scale Ray from laptop to cluster, collaborating with AI-native customers to push performance and reliability in production environments.

Responsibilities focus on improving Ray Data, scaling across pipeline stages, and building loading solutions for training workloads, with a strong emphasis on stability and fault tolerance at scale.