Software Engineer (Gen AI Solutions)

Valiant Technologies LLC

Austin (TX)

On-site

USD 96,000 - 124,000

Full time

43 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Valiant Technologies LLC in Austin, TX is seeking a Software Engineer (Gen AI Solutions) to design, build, and optimize production LLM inference services powering GenAI features. You will collaborate with ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.

The role covers designing scalable inference on Kubernetes, managing latency and cost, building CI/CD pipelines, and delivering observability and runbooks for production readiness.

Qualifications

  • 2+ years of production ML/LLM systems experience.
  • Strong Python; working knowledge of Go, C++, or Rust for performance paths.
  • Hands-on with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton.
  • Experience with PyTorch and Hugging Face ecosystem.
  • Kubernetes, Docker, and GPU scheduling experience.
  • Observability tooling: Prometheus, Grafana, OpenTelemetry.

Responsibilities

  • Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
  • Profile and reduce end-to-end latency and cost per request.
  • Apply model and serving optimizations and benchmark trade-offs between quality, latency, and throughput.
  • Build integrations with external model APIs and internal orchestration frameworks with routing, retries, and rate limiting.
  • Add observability across the request path with logging, metrics, tracing, and dashboards.
  • Set up evaluation and regression pipelines to prevent quality degradation.
  • Capacity planning, autoscaling, and GPU utilization/cost tracking.
  • Write design docs and runbooks; hand knowledge to internal teams.
  • Produce CI/CD-ready inference services and infrastructure as code.
  • Provide latency and cost benchmark reports and maintain monitoring dashboards and alerts.
  • Create architecture docs and conduct handover sessions.

Skills

Python
Go
C++
Rust
ML/LLM systems
Kubernetes
LLM serving
vLLM
TensorRT
Triton
OpenTelemetry
Prometheus
Grafana
Docker
CI/CD
GPU scheduling
REST/gRPC

Tools

Prometheus
Grafana
OpenTelemetry
Docker
TensorRT-LLM
TGI
Triton Inference Server
vLLM
ONNX Runtime

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title:Software Engineer (Gen AI Solutions)

Location: Austin, TX

Duration: 14 Months + Extension

Bill Rate: $76/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description :

We need an experienced software engineer to help design, build, and optimize production LLM inference services that power GenAI features for internal and customer-facing applications. The engineer will work with our ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.

Scope of work :

  • Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
  • Profile and reduce end-to-end latency (time to first token, tokens per second) and cost per request.
  • Apply model and serving optimizations, then benchmark the trade-offs between quality, latency, and throughput.
  • Build and maintain integrations with external model APIs and internal orchestration frameworks, including routing, fallback, retries, and rate-limit handling.
  • Add observability across the request path, covering logging, metrics, tracing, and dashboards for request volume, downstream call ratios, errors, and token usage.
  • Set up evaluation and regression pipelines so model or serving changes don't degrade output quality.
  • Do capacity planning and autoscaling, and track GPU utilization and cost.
  • Write technical design docs and runbooks, and hand knowledge over to the internal team.
  • Production-ready inference service(s) with CI/CD and infrastructure as code
  • Latency and cost benchmark reports, measured before and after optimization
  • Monitoring dashboards and alerting
  • Architecture docs, runbooks, and handover sessions

Required :

  • Software engineering experience, including 2+ years on ML/LLM systems in production
  • Strong Python, plus working knowledge of Go, C++, or Rust for performance-critical paths
  • Hands-on experience with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton Inference Server
  • KV-cache management, paged attention, prefix caching
  • Continuous/dynamic batching and speculative decoding
  • Tensor and pipeline parallelism
  • Experience with PyTorch and the Hugging Face ecosystem
  • Kubernetes, Docker, and GPU scheduling (NVIDIA GPU operator, node pools, autoscaling)
  • Integration with hosted LLM APIs (OpenAI, Gemini, Anthropic), including streaming, retries, and rate limiting
  • Observability tooling: Prometheus/Grafana, Splunk, Datadog, OpenTelemetry
  • Solid grounding in distributed systems, API design (REST/gRPC), and performance profiling

Preferred :

  • CUDA, Triton kernels, FlashAttention, or other low-level GPU optimization
  • Model compilation and runtimes: ONNX Runtime, TorchScript/torch.compile, Core ML
  • RAG pipelines and vector databases (FAISS, Milvus, pgvector)
  • LLM evaluation frameworks and building automated quality and regression benchmarks
  • Cost modeling and FinOps for GPU workloads
  • Infrastructure as code (Terraform, Helm) and CI/CD (GitHub Actions, Jenkins)
  • Contributions to open-source inference projects, or published work in ML systems
  • Can turn ambiguous requirements into concrete technical plans and trade-off proposals
  • Communicates clearly in writing (design docs, benchmark reports)
  • Works well with cross-functional teams of engineers and PMs.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Ethereum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Quantum Technologies LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Software Engineer (Gen AI Solutions)
Software Engineer (Gen AI Solutions)

Robotic Process Automation LLC • Austin (TX)

On-site
USD 86,000 - 123,000
Senior Principal Software Engineer, LLM Inference & GPU Optimization
Senior Principal Software Engineer, LLM Inference & GPU Optimization

People In AI • United States

Remote
USD 230,000 - 280,000
Bonus
Equity
Remote work within United States
AI Engineer
AI Engineer

Imigrus • Plano (TX)

On-site
USD 140,000 - 180,000
ML engineer with Gen AI
ML engineer with Gen AI

Ethereum Technologies LLC • Austin (TX)

On-site
USD 180,000 - 250,000
AI-Focused Software Developer
AI-Focused Software Developer

Ethereum Technologies LLC • Palo Alto (CA)

On-site
USD 91,000 - 124,000
AI/ML Engineer
AI/ML Engineer

Ethereum Technologies LLC • Austin (TX)

On-site
USD 95,000 - 136,000
AI/ML Engineer
AI/ML Engineer

Quantum Technologies LLC • Austin (TX)

On-site
USD 95,000 - 136,000
LLM Inference Engineer
LLM Inference Engineer

Majestic Labs ai • Los Altos (CA)

On-site
USD 210,000 - 350,000