Senior MR Multimodal LLM Scientist | Tool-Use

ByteDance; San Jose (CA); On-site; USD 212,800 - 450,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

401(k) with company match

Paid parental leave

Disability coverage

Life insurance

Wellbeing benefits

Paid holidays

Paid sick days

Paid Personal Time

Job summary

ByteDance is seeking a Senior Research Scientist for PICO in San Jose to advance multimodal large language models (MLLM) for mixed reality. You will lead architecture optimization, data construction, and end-to-end training/inference acceleration tailored to MR scenarios.

The role requires deep expertise in cross-modal fusion, tool use, and multi-turn interactions, with collaboration across software, hardware, and product teams to deliver field-ready capabilities.

Qualifications

• Master's or Ph.D. in Computer Science, Electrical Engineering, Machine Learning, Artificial Intelligence, or related field.
• Expertise in multimodal LLM pre-training, post-training, fine-tuning, or cross-modal fusion with hands-on optimization.
• Proven research in LLM tool use, reinforcement learning, LLM agents, or interactive learning.
• Proficiency in Python and C++, with experience in mainstream DL frameworks (PyTorch/TensorFlow).
• Excellent problem solving and independent research abilities in MR/MLLM tool-use integrations.

Responsibilities

• Lead R&D of multimodal LLMs for MR scenarios, optimizing architectures and end-to-end training/inference.
• Develop MLLM tool-use capabilities for MR, enabling tool calls in single- and multi-turn conversations.
• Address long-horizon, multi-turn tool-augmented tasks, including memory management and tool selection strategies.
• Stay at the forefront of MLLM, multimodal intelligence, and tool-use research; deploy innovations in PICO MR products.
• Collaborate with software, product design, and hardware teams to translate research into features.

Skills

Multimodal LLM

Python

C++

LLM tool use

Reinforcement learning

Cross-modal fusion

3D/vision understanding

Education

Master's or PhD in CS/EE/ML/AI

Tools

PyTorch

TensorFlow

3D perception toolkits

Job description

ByteDance is seeking a Senior Research Scientist for PICO in San Jose to advance multimodal large language models (MLLM) for mixed reality. You will lead architecture optimization, data construction, and end-to-end training/inference acceleration tailored to MR scenarios.

The role requires deep expertise in cross-modal fusion, tool use, and multi-turn interactions, with collaboration across software, hardware, and product teams to deliver field-ready capabilities.