GenAI Platform Architect for High-Scale LLM Inference

Socket.dev; New Jersey; On-site; USD 180,000 - 240,000; Full time

Job summary

JPMorganChase, within the Chief Data and Analytics Office, seeks a Principal Software Engineer to lead the GenAI serving platform, focusing on high-performance LLM inference, intelligent routing, and GPU efficiency. You will influence senior stakeholders, drive secure scaling, and deliver enterprise-grade AI capabilities across portfolios.

You will own design and implementation of distributed inference architectures, conduct performance benchmarking, and ensure operational excellence with robust

Qualifications

• Formal training or certification in software engineering with 7+ years experience in large-scale platforms and services.
• Strong proficiency in Python, Java, Scala, or Go with focus on code quality and testing.
• Experience leading agentic AI development practices and secure handling of data.

Responsibilities

• Design and operate a high-throughput LLM serving platform with low latency and autoscaling.
• Build GenAI Gateway / inference API for authentication, routing, and observability.
• Develop and optimize multi-backend model routing balancing quality, latency, and cost.
• Drive GPU serving optimization including memory management and KV-cache strategies.
• Implement quantization and safe rollout practices to reduce cost while preserving quality.
• Define and govern disaggregated serving architectures to improve tail latency.
• Write secure, production-grade code and review others’ work; create reusable frameworks.
• Own SDK and service integrations ensuring reliability and performance.
• Establish SLOs/SLAs and build performance benchmarking and observability.

Skills

Programming languages

Agentic AI adoption

Distributed systems

GPU inference

Security by design

Communication

SLOs/SLAs

Tools

Kubernetes

TensorRT-style

Job description

JPMorganChase, within the Chief Data and Analytics Office, seeks a Principal Software Engineer to lead the GenAI serving platform, focusing on high-performance LLM inference, intelligent routing, and GPU efficiency. You will influence senior stakeholders, drive secure scaling, and deliver enterprise-grade AI capabilities across portfolios.

You will own design and implementation of distributed inference architectures, conduct performance benchmarking, and ensure operational excellence with robust