Gen AI Inference Engineer - Scalable LLM Serving

CloudBigD

Austin (TX)

On-site

USD 86,000 - 123,000

Full time

26 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CloudBigD in Austin, TX is seeking a Software Engineer (Gen AI Solutions) to design and optimize production LLM inference services powering GenAI features. You will work with ML, platform, and product teams to accelerate model serving while reducing cost and latency, at scale.

You will implement scalable Kubernetes-based inference, enable observability, and drive CI/CD and IaC for production readiness. Candidates should have 2+ years in ML/LLM systems and strong Python with performance-minded

Qualifications

  • 2+ years building ML/LLM production systems.
  • Strong Python with performance paths in Go/C++/Rust.
  • Experience with open-weight hosting and model serving infra.

Responsibilities

  • Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
  • Profile and reduce latency and cost per request.
  • Benchmark quality, latency, and throughput trade-offs.
  • Integrate with hosted LLM APIs and internal orchestration, including routing, retries, and rate limits.
  • Add observability: logging, metrics, tracing, dashboards for usage and errors.
  • Set up evaluation/regression pipelines to protect output quality.
  • Capacity planning, autoscaling, and GPU utilization tracking.
  • Write design docs and runbooks; handover to internal team.
  • Develop CI/CD and infrastructure as code for production-ready inference services.
  • Produce latency/cost benchmark reports and monitoring dashboards.

Skills

ML/LLM in prod
Python
Go/C++/Rust

Tools

vLLM
TensorRT-LLM
TGI
SGLang
Triton Inference Server
PyTorch
Hugging Face
Kubernetes
Docker

Job description

CloudBigD in Austin, TX is seeking a Software Engineer (Gen AI Solutions) to design and optimize production LLM inference services powering GenAI features. You will work with ML, platform, and product teams to accelerate model serving while reducing cost and latency, at scale.

You will implement scalable Kubernetes-based inference, enable observability, and drive CI/CD and IaC for production readiness. Candidates should have 2+ years in ML/LLM systems and strong Python with performance-minded

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Gen AI Inference Engineer — Scalable LLM Services
Gen AI Inference Engineer — Scalable LLM Services

Ethereum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Gen AI Solutions Engineer - Production LLM Inference
Gen AI Solutions Engineer - Production LLM Inference

Robotics Technologies • Jackson (TN)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer (LLM, Kubernetes, Production)
Gen AI Inference Engineer (LLM, Kubernetes, Production)

Quantum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer (Production-Ready)
Gen AI Inference Engineer (Production-Ready)

Valiant Technologies LLC • Austin (TX)

On-site
USD 96,000 - 124,000
Gen AI Inference Engineer for Scalable LLMs
Gen AI Inference Engineer for Scalable LLMs

Robotic Process Automation LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Production ML Engineer — LLMs, Agentic AI & Inference
Production ML Engineer — LLMs, Agentic AI & Inference

INSPYR Solutions • Sunnyvale (CA)

Hybrid
USD 120,000 - 180,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Valiant Technologies LLC • Austin (TX)

On-site
USD 96,000 - 124,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

CloudBigD • Austin (TX)

On-site
USD 86,000 - 123,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Quantum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Robotics Technologies • Jackson (TN)

On-site
USD 86,000 - 123,000