Senior Real-Time Multimodal Conversational AI Scientist

Amazon; Sunnyvale (CA); On-site; USD 192,000 - 260,000; Full time

Job summary

Amazon in Sunnyvale seeks a Senior Applied Scientist to drive real-time multimodal conversational AI research and deployment. You will own a research area spanning pre-training to post-training alignment and real-time production readiness, working with inference engineers to ensure latency-aware models.

The role emphasizes foundation models for speech and audio, reward modeling, reinforcement learning, and end-to-end system design for real-time interaction.

Qualifications

• 5+ years of building machine learning models for business applications
• PhD or Master’s with 6+ years of applied research experience
• Experience programming in Java, C++, Python or related language
• Hands-on experience training large-scale foundation models
• Experience with multimodal architectures across speech, text, and audio
• Strong understanding of transformer architectures in speech/audio
• Top-tier publication record (NeurIPS, ICML, ICLR, Interspeech, ICASSP, ACL)
• Demonstrated ownership of a research area and cross-team collaboration

Responsibilities

• Build and train large-scale multimodal foundation models for real-time speech and audio
• Advance scaling and efficiency of conversational models with data-model-performance tradeoffs
• Design architectures considering hardware constraints for real-time deployment
• Develop training methodologies for real-time multimodal processing
• Contribute to state-of-the-art patterns in efficient architectures and training methods
• Design and build reward models for speech systems with real-time responsiveness
• Apply RL to shape conversational behavior and timing
• Develop post-training pipelines from SFT through RL alignment for multimodal outputs
• Create evaluation frameworks for latency, audio quality, and naturalness
• Advance real-time perception to process audio while generating responses
• Develop techniques for concurrent input/output with human-like timing

Skills

Large-scale models training

Multimodal architectures

Transformer for speech/audio

RLHF/RL training

Java/C++/Python

Research ownership

Production-ready systems

Education

PhD

Master’s +6 years applied research

Job description

Amazon in Sunnyvale seeks a Senior Applied Scientist to drive real-time multimodal conversational AI research and deployment. You will own a research area spanning pre-training to post-training alignment and real-time production readiness, working with inference engineers to ensure latency-aware models.

The role emphasizes foundation models for speech and audio, reward modeling, reinforcement learning, and end-to-end system design for real-time interaction.