Senior LLM Inference Engineer – AI Platform Lead

Fairygodboss; Jersey City (NJ); On-site; USD 190,000 - 240,000; Full time

Job summary

JPMorganChase is seeking a Principal Software Engineer to own LLM inference performance and optimization for scalable AI platforms. You will benchmark, quantify improvements, and drive decisions with leaders to improve speed, cost efficiency, and production readiness across models.

You will collaborate with EKS and disaggregated serving teams, lead quantization experiments, and shape guardrails for secure and auditable AI workflows.

Qualifications

• Formal training or certification on software engineering concepts and 7+ years applied experience.
• Deep, hands-on experience with LLM inference systems - vLLM, TensorRT-LLM, SGLang, LLM-D or equivalent production serving engines.
• Strong grasp of GPU memory architecture and quantization at inference time.
• Experience with quantization techniques and real-world tradeoffs at scale.
• Familiarity with speculative decoding and acceptance-rate variables in production workloads.
• Rigorous benchmarking instincts with data-backed claims.

Responsibilities

• Own systematic benchmarking and performance characterization across all production LLM workloads.
• Design and execute quantization experiments - FP8, INT8/INT4, measuring accuracy delta, throughput, memory, cost per token.
• Drive speculative decoding strategy across the model portfolio and provide configuration recommendations.
• Build and maintain a GPU efficiency scorecard for leadership review.
• Benchmark platform against external providers and published industry numbers.
• Lead inference engine upgrade evaluations and validate before production promotion.
• Collaborate on KV-cache optimization, prefix caching, and multi-node serving architecture.
• Design and run GPU chaos engineering, monitoring, and recovery measurement.
• Architect AI-enabled engineering workflows with guardrails for validation, security, resiliency.
• Apply SDLC toolchain and enterprise AI capabilities to improve automation and quality.

Skills

LLM inference systems

GPU memory architecture

quantization techniques

speculative decoding

benchmarking

cloud GPU infrastructure

stakeholder communication

security/resiliency governance

Tools

vLLM

TensorRT-LLM

SGLang

LLM-D

GPTQ/AWQ

Job description

JPMorganChase is seeking a Principal Software Engineer to own LLM inference performance and optimization for scalable AI platforms. You will benchmark, quantify improvements, and drive decisions with leaders to improve speed, cost efficiency, and production readiness across models.

You will collaborate with EKS and disaggregated serving teams, lead quantization experiments, and shape guardrails for secure and auditable AI workflows.