AI Infra Engineer: Build Scalable Model Serving

Deep Infra Inc.; Palo Alto (CA); On-site; USD 150,000 - 195,000; Full time

Benefits offered by this job

Open-source contributions

C++/CUDA experience

Job summary

DeepInfra is building the infrastructure layer for AI, serving trillions of tokens weekly across many workloads. We are looking for strong Software Engineers to join our Palo Alto team and help design, build, and scale infrastructure to serve open-source AI models in production.

You’ll own end-to-end features, implement and optimize models with Python, C++, CUDA, NCCL, and operate production model-serving systems with monitoring.

Qualifications

• Bachelor’s or Master’s degree in CS, CE, or related field.
• 1–4 years of relevant experience in software/AI.
• Strong fundamentals in data structures, algorithms, and software design.
• Proficiency in Python and AI/ML frameworks (PyTorch, TensorFlow).

Responsibilities

• Design, develop, and test inference solutions for state-of-the-art AI models.
• Implement, optimize, and evaluate AI models with Python, C++, CUDA, NCCL.
• Own and operate production model-serving systems with monitoring and debugging.
• Build new features, improve system performance, and contribute to design.
• Participate in code reviews and technical discussions to maintain high engineering standards.
• Explore AI/ML techniques to improve model performance and efficiency.
• Take ideas from concept to production.

Skills

Data structures

Algorithms

Software design

Debugging

Communication

Independent work

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Python

C++

CUDA

NCCL

PyTorch

TensorFlow

Git

Job description

DeepInfra is building the infrastructure layer for AI, serving trillions of tokens weekly across many workloads. We are looking for strong Software Engineers to join our Palo Alto team and help design, build, and scale infrastructure to serve open-source AI models in production.

You’ll own end-to-end features, implement and optimize models with Python, C++, CUDA, NCCL, and operate production model-serving systems with monitoring.