Senior Deployment Engineer (LLM Inference & Optimization)

Coreweave

Sunnyvale (CA)

On-site

USD 180,000 - 240,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CoreWeave is seeking a Deployment Engineer to join its AI infrastructure team in a hands-on role focused on production-grade model deployment and optimization. You will work directly with customers to design, benchmark, and deploy AI models, improve performance, and integrate solutions with enterprise systems.

You will own technical delivery from discovery to deployment, collaborating with product and engineering teams to meet defined targets for quality, latency, throughput, and cost.

Qualifications

  • Experience with model-serving frameworks such as vLLM, SGLang, TensorRT-LLM, or similar technologies.
  • Experience optimizing embedding models, rerankers, vision models, or multimodal systems.
  • Experience improving retrieval quality and evaluating end-to-end RAG applications.
  • 5+ years of experience in software engineering, ML engineering, AI infrastructure, or a related hands-on technical role.
  • Contributions to open-source model-serving, optimization, or evaluation projects.
  • Familiarity with GPU-based cloud environments, containers, and monitoring tools used to operate model deployments.
  • Experience delivering enterprise AI implementations or customer-facing technical engagements.
  • Experience building evaluations and benchmarks to measure model quality, latency, throughput, and resource efficiency.
  • The ability to work directly with customers, communicate technical tradeoffs clearly, and translate business requirements into measurable engineering targets.
  • Experience developing APIs and integrating model endpoints with production applications and customer data systems.
  • Strong Python programming skills and experience with frameworks such as PyTorch, Hugging Face Transformers, or comparable tools.
  • Strong ownership and troubleshooting skills, with the ability to move from an ambiguous problem to a tested solution.
  • An understanding of GPU memory constraints, numerical precision, batching, caching, and parallelism, including how these affect inference performance.
  • Hands-on experience deploying and optimizing AI models in production, particularly LLMs.
  • Experience profiling GPU workloads using CUDA, NVIDIA Nsight, or related tools.
  • Experience with parameter-efficient fine-tuning techniques such as LoRA or QLoRA.
  • Experience with speculative decoding, multi-GPU inference, or distributed model serving.

Responsibilities

  • Lead technical discovery to define use cases, workloads, infrastructure constraints, and targets for model quality, latency, throughput, and cost.
  • Publish and explain benchmark findings to customers; guide deployment and optimization decisions.
  • Lead workshops and customer engagements to ensure successful technical delivery.
  • Troubleshoot performance issues across model execution, GPU utilization, memory, serving infra, and app behavior.
  • Build reusable evaluation tools, benchmark harnesses, deployment configurations, and runbooks; feed insights back to product/engineering.

Skills

AI engineering
Model optimization
Customer-facing
Discovery & benchmarking
Deployment
Enterprise integration
Coding
Performance bottlenecks
Model quality
Latency optimization
Throughput optimization
Cost optimization
Prompt optimization
RAG concepts
Fine-tuning
Retrieval quality
Production readiness
API integration
Production apps
Python
PyTorch
HF Transformers
GPU memory
Numerical precision
Batching
Caching
Parallelism
LoRA
QLoRA
Speculative decoding
Multi-GPU inference
Distributed serving

Tools

vLLM
SGLang
TensorRT-LLM
CUDA

Job description

  • We're looking for a Deployment Engineer who combines hands-on AI engineering, model optimization, and strong customer-facing skills
  • You'll work directly with customers to bring AI models and applications into production, improve their performance, and integrate them with enterprise systems
  • You'll own technical delivery from discovery and benchmarking through deployment and adoption, working closely with customer stakeholders and our product and engineering teams
  • This role requires someone who enjoys writing code, experimenting with models, diagnosing performance bottlenecks, and helping customers achieve measurable improvements in model quality, latency, throughput, and cost
  • Understand customer requirements: Lead technical discovery to define use cases, workloads, infrastructure constraints, and targets for model quality, latency, throughput, and cost
  • Optimize inference: Apply techniques such as quantization, batching, prefix caching, KV-cache optimization, and parallelism to improve performance while preserving output quality
  • Adapt models to customer needs: Improve results through prompt optimization, retrieval-augmented generation, and fine-tuning where appropriate. Evaluate the impact of each approach
  • Troubleshoot performance: Investigate issues across model execution, GPU utilization, memory consumption, serving infrastructure, and application behavior
  • Lead technical customer engagement: Run workshops, present benchmark findings, explain tradeoffs, and guide customers through deployment and optimization decisions
  • Build reusable capabilities: Create evaluation tools, benchmark harnesses, deployment configurations, and runbooks. Bring recurring customer needs and performance findings back to product and engineering
What success looks like
  • You help customers deploy AI solutions that meet agreed quality and performance targets
  • Your work produces measurable improvements in latency, throughput, GPU utilization, and cost while maintaining model quality
  • Customers understand the tradeoffs, trust your technical guidance, and can operate the solutions you deliver
  • Your findings and tools make future deployments faster and more effective
  • Experience with model-serving frameworks such as vLLM, SGLang, TensorRT-LLM, or similar technologies
  • Experience optimizing embedding models, rerankers, vision models, or multimodal systems
  • Experience improving retrieval quality and evaluating end-to-end RAG applications
  • 5+ years of experience in software engineering, ML engineering, AI infrastructure, or a related hands-on technical role
  • Contributions to open-source model-serving, optimization, or evaluation projects
  • Familiarity with GPU-based cloud environments, containers, and monitoring tools used to operate model deployments
  • Experience delivering enterprise AI implementations or customer-facing technical engagements
  • Experience building evaluations and benchmarks to measure model quality, latency, throughput, and resource efficiency
  • The ability to work directly with customers, communicate technical tradeoffs clearly, and translate business requirements into measurable engineering targets
  • Experience developing APIs and integrating model endpoints with production applications and customer data systems
  • Strong Python programming skills and experience with frameworks such as PyTorch, Hugging Face Transformers, or comparable tools
  • Strong ownership and troubleshooting skills, with the ability to move from an ambiguous problem to a tested solution
  • An understanding of GPU memory constraints, numerical precision, batching, caching, and parallelism, including how these affect inference performance
  • Hands-on experience deploying and optimizing AI models in production, particularly LLMs
  • Experience profiling GPU workloads using CUDA, NVIDIA Nsight, or related tools
  • Experience with parameter-efficient fine-tuning techniques such as LoRA or QLoRA
  • Experience with speculative decoding, multi-GPU inference, or distributed model serving
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Applied Scientist, Efficient LLM Inference & Model Optimization
Senior Applied Scientist, Efficient LLM Inference & Model Optimization

Nebius Group • United States

Remote
USD 180,000 - 260,000
Senior LLMOps Engineer
Senior LLMOps Engineer

UNAVAILABLE • McLean (VA)

On-site
USD 180,000 - 240,000
Machine Learning Engineer, LLM Inference Optimization
Machine Learning Engineer, LLM Inference Optimization

GMI Cloud, Inc • San Francisco (CA)

On-site
USD 180,000 - 280,000
AI Engineer / LLM Systems Engineer
AI Engineer / LLM Systems Engineer

Sphere Software • United States

On-site
USD 120,000 - 170,000
LLMOps Engineer
LLMOps Engineer

UNAVAILABLE • McLean (VA)

On-site
USD 150,000 - 210,000
AI Engineer - VA
AI Engineer - VA

LawPro.ai • Virginia

On-site
USD 150,000 - 190,000
AI/ML Engineer – (Next-Generation AI Platforms & Workloads)
AI/ML Engineer – (Next-Generation AI Platforms & Workloads)

Veear Projects Inc. • Sunnyvale (CA)

On-site
USD 140,000 - 210,000
Senior AI Platform Engineer
Senior AI Platform Engineer

Code Metal • United States

Hybrid
USD 150,000 - 190,000
Health care plan (Medical, Dental &amp
401k/IRA Retirement plan
Flexible hybrid work arrangement
+2
Senior Principal Software Engineer, LLM Inference & GPU Optimization
Senior Principal Software Engineer, LLM Inference & GPU Optimization

People In AI • United States

Remote
USD 230,000 - 280,000
Bonus
Equity
Remote work within United States
Machine Learning Engineer- Inference Optimization | Experienced Hire
Machine Learning Engineer- Inference Optimization | Experienced Hire

Susquehanna International Group, LLP • Bala Cynwyd (PA)

On-site
USD 150,000 - 230,000