Gen AI Inference Engineer (Production-Ready)

Valiant Technologies LLC

Austin (TX)

On-site

USD 96,000 - 124,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Valiant Technologies LLC in Austin, TX is seeking a Software Engineer (Gen AI Solutions) to design, build, and optimize production LLM inference services powering GenAI features. You will collaborate with ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.

The role covers designing scalable inference on Kubernetes, managing latency and cost, building CI/CD pipelines, and delivering observability and runbooks for production readiness.

Qualifications

  • 2+ years of production ML/LLM systems experience.
  • Strong Python; working knowledge of Go, C++, or Rust for performance paths.
  • Hands-on with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton.
  • Experience with PyTorch and Hugging Face ecosystem.
  • Kubernetes, Docker, and GPU scheduling experience.
  • Observability tooling: Prometheus, Grafana, OpenTelemetry.

Responsibilities

  • Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
  • Profile and reduce end-to-end latency and cost per request.
  • Apply model and serving optimizations and benchmark trade-offs between quality, latency, and throughput.
  • Build integrations with external model APIs and internal orchestration frameworks with routing, retries, and rate limiting.
  • Add observability across the request path with logging, metrics, tracing, and dashboards.
  • Set up evaluation and regression pipelines to prevent quality degradation.
  • Capacity planning, autoscaling, and GPU utilization/cost tracking.
  • Write design docs and runbooks; hand knowledge to internal teams.
  • Produce CI/CD-ready inference services and infrastructure as code.
  • Provide latency and cost benchmark reports and maintain monitoring dashboards and alerts.
  • Create architecture docs and conduct handover sessions.

Skills

Python
Go
C++
Rust
ML/LLM systems
Kubernetes
LLM serving
vLLM
TensorRT
Triton
OpenTelemetry
Prometheus
Grafana
Docker
CI/CD
GPU scheduling
REST/gRPC

Tools

Prometheus
Grafana
OpenTelemetry
Docker
TensorRT-LLM
TGI
Triton Inference Server
vLLM
ONNX Runtime

Job description

Valiant Technologies LLC in Austin, TX is seeking a Software Engineer (Gen AI Solutions) to design, build, and optimize production LLM inference services powering GenAI features. You will collaborate with ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.

The role covers designing scalable inference on Kubernetes, managing latency and cost, building CI/CD pipelines, and delivering observability and runbooks for production readiness.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Gen AI Solutions Engineer - Production LLM Inference
Gen AI Solutions Engineer - Production LLM Inference

Robotics Technologies • Jackson (TN)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer - Scalable LLM Serving
Gen AI Inference Engineer - Scalable LLM Serving

CloudBigD • Austin (TX)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer (LLM, Kubernetes, Production)
Gen AI Inference Engineer (LLM, Kubernetes, Production)

Quantum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer — Scalable LLM Services
Gen AI Inference Engineer — Scalable LLM Services

Ethereum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Gen AI Inference Engineer for Scalable LLMs
Gen AI Inference Engineer for Scalable LLMs

Robotic Process Automation LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Senior AI/ML Engineer: Lead Generative AI in Production
Senior AI/ML Engineer: Lead Generative AI in Production

Valiant Technologies LLC • Boston (MA)

On-site
USD 190,000 - 270,000
AI Engineer: Generative AI, LLMs & NLP Specialist
AI Engineer: Generative AI, LLMs & NLP Specialist

Valiant Technologies LLC • Raleigh (NC)

On-site
USD 120,000 - 180,000
AI Platform Architect & Real-Time Inference Leader
AI Platform Architect & Real-Time Inference Leader

Valiant Technologies LLC • Concord (CA)

On-site
USD 83,000 - 113,000
Senior AI Platform Engineer — Production LLM Ops
Senior AI Platform Engineer — Production LLM Ops

Valiant Technologies LLC • Boston (MA)

On-site
USD 78,000 - 112,000
Production ML Engineer — LLMs, Agentic AI & Inference
Production ML Engineer — LLMs, Agentic AI & Inference

INSPYR Solutions • Sunnyvale (CA)

Hybrid
USD 120,000 - 180,000