Senior Real-Time Multimodal Inference Engineer

Amazon; Boston (MA); On-site; USD 168,000 - 227,000; Full time

Job summary

Amazon is seeking a Senior Inference Engineer to own end-to-end inference for real-time multimodal conversational AI in a full-stack role. You will shape model architectures for servability, build real-time runtimes, and create offline systems for training and reinforcement learning.

You will work across research, engineering, and hardware teams to ensure sub-second latency while managing cost. You will co-design architectures with scientists, optimize KV-cache and attention, and drive efficient

Qualifications

• 5+ years of professional software development experience.
• 5+ years of programming with at least one programming language.
• 4+ years of leading design or architecture of new and existing systems.
• Bachelor's degree in computer science or equivalent.
• Experience as a mentor or tech lead.
• 2+ years of hands-on experience optimizing inference for neural models.
• Strong understanding of deep learning architectures and multimodal domains.
• Production experience delivering latency-constrained, real-time inference systems.

Responsibilities

• Partner with research scientists to surface latency, memory, and cost implications of architecture choices.
• Implement and optimize the inference path for large-scale multimodal models.
• Apply efficiency techniques across the stack and quantify latency trade-offs.
• Develop high-performance kernels for critical operations and integrate into production serving.
• Profile end-to-end performance and identify bottlenecks in large-scale inference workloads.
• Own the real-time serving path for streaming multimodal conversational AI.
• Build and tune batching, scheduling, and preemption for throughput vs latency.
• Customize production serving frameworks for real-time streaming generative models.
• Implement multi-GPU inference and drive costs toward production baselines.
• Establish latency, throughput, and cost benchmarks for deployment.

Skills

Inference engineering

Latency optimization

GPU optimization

Python

C++

Education

Bachelor's degree in CS or equivalent

Tools

Nsight Compute

PyTorch

vLLM

CUDA

Job description

Amazon is seeking a Senior Inference Engineer to own end-to-end inference for real-time multimodal conversational AI in a full-stack role. You will shape model architectures for servability, build real-time runtimes, and create offline systems for training and reinforcement learning.

You will work across research, engineering, and hardware teams to ensure sub-second latency while managing cost. You will co-design architectures with scientists, optimize KV-cache and attention, and drive efficient