Senior LLM Inference Engineer, Performance Lead

JP Morgan Chase

Jersey City (NJ)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

JPMorganChase is seeking a Principal Software Engineer to lead LLM inference optimization for year-scale deployment in enterprise AI platforms. You will own benchmarking, quantization experiments, and speculative decoding strategies across a multi-model portfolio while partnering with EKS-based serving teams.

You will influence architectural decisions, drive efficiency, and ensure secure, auditable AI workflows with guardrails for governance and reliability in a high-stakes financial environment.

Qualifications

  • 7+ years of software engineering experience.
  • Deep hands-on experience with LLM inference systems.
  • Strong understanding of GPU memory and quantization.
  • Experience with quantization techniques and their tradeoffs at scale.
  • Familiarity with speculative decoding and production acceptance rates.
  • Rigorous benchmarking with measurable results.
  • Comfort operating in cloud GPU infrastructure (AWS/EKS).
  • Ability to communicate trade-offs to senior stakeholders.
  • Experience designing agentic AI-enabled development practices and governance.
  • Strong understanding of responsible AI and security implications.

Responsibilities

  • Own systematic benchmarking and performance characterization across all production LLM workloads.
  • Design and execute quantization experiments - FP8, INT8/INT4 (GPTQ/AWQ), next-generation formats.
  • Drive speculative decoding strategy across the model portfolio and provide configuration recommendations.
  • Build and maintain a GPU efficiency scorecard for leadership visibility.
  • Benchmark platform against external providers and industry numbers to close gaps.
  • Lead inference engine upgrade evaluations and architecture choices.
  • Collaborate on KV-cache optimization and multi-node serving architecture.
  • Design GPU chaos engineering experiments and monitor hardware health.
  • Architect agentic AI-enabled workflows with guardrails and auditability.
  • Apply SDLC tools and AI-assisted development capabilities to scale automation.

Skills

LLM inference experience
Quantization techniques
GPU memory architecture
Benchmarking
Speculative decoding
Cloud infrastructure (AWS/EKS)
Agentic AI-enabled development
Security/resiliency governance
Communication with leadership
Performance optimization

Tools

vLLM
TensorRT-LLM
SGLang
LLM-D

Job description

JPMorganChase is seeking a Principal Software Engineer to lead LLM inference optimization for year-scale deployment in enterprise AI platforms. You will own benchmarking, quantization experiments, and speculative decoding strategies across a multi-model portfolio while partnering with EKS-based serving teams.

You will influence architectural decisions, drive efficiency, and ensure secure, auditable AI workflows with guardrails for governance and reliability in a high-stakes financial environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal LLM Inference Engineer — AI Platform
Principal LLM Inference Engineer — AI Platform

JP Morgan Chase • Jersey City (NJ)

On-site
USD 210,000 - 320,000
Principal Software Engineer - LLM Optimization
Principal Software Engineer - LLM Optimization

JPMorgan Chase & Co. • Jersey City (NJ)

On-site
USD 180,000 - 240,000
Senior Principal LLM Engineer — AI Platform & Scale
Senior Principal LLM Engineer — AI Platform & Scale

JPMorgan Chase & Co. • Palo Alto (CA)

On-site
USD 233,000 - 325,000
Principal Software Engineer - LLM Optimization
Principal Software Engineer - LLM Optimization

JP Morgan Chase • Jersey City (NJ)

On-site
USD 210,000 - 320,000
Principal Software Engineer - LLM Optimization
Principal Software Engineer - LLM Optimization

JP Morgan Chase • Jersey City (NJ)

On-site
USD 180,000 - 240,000
Senior Principal AI/ML Architect — LLM & GNN Platforms
Senior Principal AI/ML Architect — LLM & GNN Platforms

Testing Site - JPMC • Palo Alto (CA)

On-site
USD 250,000 - 330,000
Health insurance
Retirement plan
On-site facilities and amenities
Senior ML Engineer: LLMs & Production AI for Finance
Senior ML Engineer: LLMs & Production AI for Finance

JP Morgan Chase • Palo Alto (CA)

On-site
USD 180,000 - 240,000
Senior Principal AI/ML Engineer, LLM & GNN Scale
Senior Principal AI/ML Engineer, LLM & GNN Scale

JP Morgan Chase • Palo Alto (CA)

On-site
USD 180,000 - 240,000
Senior ML Engineer - LLMs & Production AI for Finance
Senior ML Engineer - LLMs & Production AI for Finance

JPMorgan Chase & Co. • Palo Alto (CA)

On-site
USD 150,000 - 230,000
Senior AI/ML Engineer – LLMs & Cloud Solutions Lead
Senior AI/ML Engineer – LLMs & Cloud Solutions Lead

JPMorgan Chase & Co. • Jersey City (NJ)

On-site
USD 150,000 - 210,000