Senior Research Scientist (Multimodal Large Language Model) - PICO

ByteDance; San Jose (CA); On-site; USD 212,800 - 450,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

401(k) with company match

Paid parental leave

Disability coverage

Life insurance

Wellbeing benefits

Paid holidays

Paid sick days

Paid Personal Time

Job summary

ByteDance is seeking a Senior Research Scientist for PICO in San Jose to advance multimodal large language models (MLLM) for mixed reality. You will lead architecture optimization, data construction, and end-to-end training/inference acceleration tailored to MR scenarios.

The role requires deep expertise in cross-modal fusion, tool use, and multi-turn interactions, with collaboration across software, hardware, and product teams to deliver field-ready capabilities.

Qualifications

• Master's or Ph.D. in Computer Science, Electrical Engineering, Machine Learning, Artificial Intelligence, or related field.
• Expertise in multimodal LLM pre-training, post-training, fine-tuning, or cross-modal fusion with hands-on optimization.
• Proven research in LLM tool use, reinforcement learning, LLM agents, or interactive learning.
• Proficiency in Python and C++, with experience in mainstream DL frameworks (PyTorch/TensorFlow).
• Excellent problem solving and independent research abilities in MR/MLLM tool-use integrations.

Responsibilities

• Lead R&D of multimodal LLMs for MR scenarios, optimizing architectures and end-to-end training/inference.
• Develop MLLM tool-use capabilities for MR, enabling tool calls in single- and multi-turn conversations.
• Address long-horizon, multi-turn tool-augmented tasks, including memory management and tool selection strategies.
• Stay at the forefront of MLLM, multimodal intelligence, and tool-use research; deploy innovations in PICO MR products.
• Collaborate with software, product design, and hardware teams to translate research into features.

Skills

Multimodal LLM

Python

C++

LLM tool use

Reinforcement learning

Cross-modal fusion

3D/vision understanding

Education

Master's or PhD in CS/EE/ML/AI

Tools

PyTorch

TensorFlow

3D perception toolkits

Job description

Senior Research Scientist (Multimodal Large Language Model) - PICO

Location: San Jose

Team: Technology

Employment Type: Regular

Job Code: A57637

About the Team

PICO-MR team is dedicated to pioneering core technologies for intelligent human-computer interaction in MR environments, with a focus on integrating multimodal large language models (MLLM) and tool-use capabilities to redefine user experiences. Our R&D directions cover cutting-edge fields including multimodal scene understanding, MLLM-based agent systems, tool-augmented MR interaction, 3D environment perception, and AIGC-driven content generation. Within MR scenarios, our work spans MLLM optimization and adaptation for MR, intelligent task execution with tool use, multimodal scene understanding (vision, point clouds, text), AIGC-based scene generation, depth estimation (Mono/Stereo/MVS), 3D environment perception, large-scale 3D scene reconstruction (3DGS, NeRF, etc.), visual localization, and lighting estimation - encompassing both fundamental research breakthroughs and industrial-grade solution deployment.

Responsibilities

• Lead the R&D of multimodal large language models (MLLM) tailored for MR scenarios, integrating vision, point clouds, text, and other multimodal information—including model architecture optimization, cross-modal alignment, data construction, evaluation system enhancement, and end-to-end training/inference acceleration.
• Drive the research and implementation of MLLM tool-use capabilities in MR environments, enabling models to proficiently utilize spatial interaction and spatial computing-related professional tools, support tool calls for both single-turn and multi-turn conversations, and solve complex user tasks through interaction.
• Address key challenges in long-horizon, multi-turn tool-augmented tasks in MR, such as context memory management, tool selection strategy, and error correction mechanisms.
• Keep abreast of cutting-edge technologies in MLLM, multimodal intelligence, and tool-use research, and lead the application and deployment of innovative technologies in PICO's MR products.
• Collaborate with cross-functional teams (including software engineering, product design, and hardware development) to translate research outcomes into practical features that enhance user experience.

Qualifications

Minimum Qualifications

• Master's or Ph.D. degree in Computer Science, Electrical Engineering, Machine Learning, Artificial Intelligence, or a related quantitative field.
• Expertise in multimodal large model pre-training, post-training, fine-tuning, or cross-modal fusion technologies, with hands-on experience in model optimization, training workflow design, and performance tuning.
• Proven research experience in LLM tool use, reinforcement learning, LLM agents, or interactive learning, with a deep understanding of single-turn and multi-turn interaction mechanisms.
• Proficiency in core 2D/3D computer vision tasks, including detection, segmentation, depth estimation, image matching, and 3D scene perception.
• Skilled in Python and C++, with solid programming capabilities and experience in developing large-scale models using mainstream deep learning frameworks (PyTorch/TensorFlow).
• Excellent problem-solving and independent research abilities, capable of addressing complex technical challenges in the integration of MR and MLLM tool use.

Preferred Qualifications

• Publications in AI/ML/CV conferences (e.g., NeurIPS, ICML, ICLR, CVPR, ICCV, ECCV, ACL, EMNLP) focusing on multimodal large models, LLM tool use, or agent systems.
• Hands-on experience in building large-scale MLLM training pipelines, tool-use evaluation systems, or multimodal agent platforms.
• Familiarity with MR/AR/VR technologies, spatial computing, or 3D scene reconstruction (3DGS, NeRF, etc.) is a strong plus.
• Experience in addressing long-horizon reasoning or asynchronous agent behavior challenges is highly valued.
• Award winners of competitions such as ACM-ICPC, NOI/IOI, TopCoder, or AI/ML contests (e.g., Kaggle) are preferred.
• Strong collaboration and communication skills, able to lead research initiatives and drive cross-team technical alignment.

Job Information

The base salary range for this position in the selected city is $212,800 - $450,000 annually.

Compensation may vary outside of this range depending on a number of factors, including a candidate’s qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.

Benefits

Employees have day-one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).

The Company reserves the right to modify or change these benefits programs at any time, with or without notice.

For Los Angeles County (unincorporated) Candidates

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state, and local laws including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Our company believes that criminal history may have a direct, adverse and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment:

• Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues.
• Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems.
• Exercising sound judgment.

Reasonable Accommodation

ByteDance is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/RA-request