Sr. Software Engineer

Spectro Cloud; San Jose (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Spectro Cloud in San Jose, CA, seeks a senior engineer to own AI inference at scale. You will build production systems to serve models across data centers and edge sites, ensuring speed, reliability, and efficiency.

Bring new models into service, improve performance under real demand, and set testing and release standards. Strong collaboration across teams is required, with Kubernetes experience highly desired.

Qualifications

• Hands-on experience running AI or LLM inference in production.
• Working understanding of inference engines, batching, and memory management.
• Experience improving performance, reliability, and efficiency of production systems.
• Experience with distributed systems and multi-GPU workloads.
• Experience designing benchmarks and release criteria.
• Bachelor's degree in Computer Science or related field.
• 4+ years of software development experience, or 2+ years with a Master's degree.
• Strong Python and hands-on PyTorch.
• Golang for backend services and APIs.
• Working knowledge of Kubernetes.
• Experience with modern GPU platforms, drivers, and monitoring tools.
• Understanding of high-speed networking and interconnects.
• Experience with cloud-native applications on AWS, Azure, or GCP.
• Comfort in Linux environments and with common command line tools.
• Experience with automated testing.
• Familiarity with AI-assisted development tools.
• Clear written communication in English.
• Experience with model optimization, quantization, and acceleration.
• GPU scheduling in Kubernetes
• Experience with monitoring and observability tools
• Kubernetes certification such as CKA or CKAD

Responsibilities

• Build and operate AI inference services in production, with engines such as vLLM, SGLang, or TensorRT-LLM.
• Bring new models into service, from first test to full rollout.
• Improve throughput, response time, and memory use across the stack.
• Find the root cause of performance and reliability issues, then fix them.
• Design benchmarks and release criteria for every model we ship.
• Set reliability targets and meet them in production.
• Scale inference across many GPUs and nodes.
• Run inference on modern GPU platforms, including the latest accelerators.
• Automate deployment and operations on Kubernetes.
• Write runbooks, tuning guides, and clear technical documentation

Skills

Python

PyTorch

Kubernetes

Golang

Distributed systems

Multi-GPU

Model optimization

Education

Bachelor's degree in Computer Science or related field

Tools

CUDA/TensorRT

Linux

Job description

Who We Are

Spectro Cloud helps platform teams and cloud providers modernize and manage infrastructure for the AI era without adding tools or operational complexity. PaletteAI gives enterprises, public sector organizations, and service providers one platform to build, deploy, manage, and scale full-stack environments across virtual machines, Kubernetes, edge, and AI infrastructure. PaletteAI’s Inference Launchpad extends it to local model serving, with open models running on day one.

Who We Are

Spectro Cloud helps platform teams and cloud providers modernize and manage infrastructure for the AI era without adding tools or operational complexity. PaletteAI gives enterprises, public sector organizations, and service providers one platform to build, deploy, manage, and scale full-stack environments across virtual machines, Kubernetes, edge, and AI infrastructure. PaletteAI’s Inference Launchpad extends it to local model serving, with open models running on day one. Founded by the team behind Palette, a leading Kubernetes management platform, and backed by leading investors, Spectro Cloud serves customers worldwide, including regulated and air-gapped environments.

About The Job

At Spectro Cloud, we are looking for a senior engineer to own AI inference at scale. Our customers run models on infrastructure they control, from data centers to edge sites. You will build the systems that serve those models in production and keep them fast, reliable, and efficient. You will bring new models into service, improve how they perform under real demand, and set the standard for how we test and release them. Your work will shape how our customers use AI in production. We are looking for an engineer who enjoys difficult problems, works well across teams, and can turn a deep technical issue into a clear plan. Experience with Kubernetes is highly desired.

Qualities

As a software engineer at Spectro Cloud, you will succeed by embracing adaptability, tackling complex challenges, and navigating dynamic environments with confidence. You can break down intricate problems into manageable steps, deliver value iteratively, and explain your work clearly to the people who depend on it. You take ownership of outcomes and see work through to production. At Spectro Cloud, we champion innovation and collaboration, fostering a culture where shared ideas drive progress.

Key Responsibilities

• Build and operate AI inference services in production, with engines such as vLLM, SGLang, or TensorRT-LLM
• Bring new models into service, from first test to full rollout
• Improve throughput, response time, and memory use across the stack
• Find the root cause of performance and reliability issues, then fix them
• Design benchmarks and release criteria for every model we ship
• Set reliability targets and meet them in production
• Scale inference across many GPUs and nodes
• Run inference on modern GPU platforms, including the latest accelerators
• Automate deployment and operations on Kubernetes
• Write runbooks, tuning guides, and clear technical documentation

Qualifications

While we don't expect candidates to check every box, we do look for the following when assessing applicants:

• Hands-on experience running AI or LLM inference in production, with an engine such as vLLM or TensorRT-LLM
• A working understanding of inference engines, including batching, caching, and memory management
• Experience improving the performance, reliability, and efficiency of production systems
• Experience with distributed systems and multi-GPU workloads
• Experience designing benchmarks and release criteria
• Bachelor's degree in Computer Science or a related field
• 4+ years of software development experience, or 2+ years with a Master's degree
• Strong Python and hands-on PyTorch
• Golang for backend services and APIs
• Working knowledge of Kubernetes
• Experience with modern GPU platforms, drivers, and monitoring tools
• Understanding of high-speed networking and interconnects
• Experience with cloud-native applications on AWS, Azure, or GCP
• Comfort in Linux environments and with common command line tools
• Experience with automated testing
• Familiarity with AI-assisted development tools
• Clear written communication in English
• Experience with model optimization, quantization, and acceleration
• GPU scheduling in Kubernetes
• Experience with monitoring and observability tools
• Kubernetes certification such as CKA or CKAD

The Hiring Process

At Spectro Cloud, we highly value your time and dedication throughout the hiring process. Therefore, we aim to ensure that our interactions are efficient and focused on delivering value. Our interview process for Engineering roles typically comprises three to four stages:

• Initial screening interview
• One or two technical interviews, including hands-on coding assessments
• Final round to assess team fit and engage in detailed discussions

The majority of our interviews are conducted via Zoom. We recommend having a laptop with a reliable internet connection, and if possible, a functional camera for a more engaging experience. We appreciate your commitment to our hiring process and look forward to getting to know you better.