Senior Research Scientist, Multimodal Foundation Models and Robotics

NVIDIA Corporation; Santa Clara (CA); On-site; USD 184,000 - 357,000; Full time

Job summary

An established industry player is seeking a Senior Research Scientist to innovate in the field of Multimodal Foundation Models and Robotics. This role involves designing cutting-edge AI algorithms and deploying them in both virtual and physical environments. You will collaborate with a talented team dedicated to creating general-purpose embodied agents that push the boundaries of technology. Your work will significantly influence groundbreaking research projects and product development. If you're passionate about AI and robotics, this is an exciting opportunity to make a lasting impact in a forward-thinking organization.

Benefits

Equity

Comprehensive Benefits

Diversity and Inclusion Initiatives

Qualifications

• 5+ years of experience in multimodal foundation models and robotics.
• Outstanding engineering skills in model training frameworks.

Responsibilities

• Design and implement AI algorithms for humanoid robots.
• Develop AI training and inference methods for foundation models.

Skills

Python

C++

CUDA

Reinforcement Learning

Imitation Learning

Model Training Frameworks

Large-scale Machine Learning

Robot Kinematics

Robot Dynamics

Control Methods

Education

Ph.D. in Computer Science/Engineering

Equivalent Research Experience

Tools

PyTorch

Jax

TensorFlow

ROS

MuJoCo

Isaac Sim

Job description

Senior Research Scientist, Multimodal Foundation Models and Robotics

Senior Research Scientist, Multimodal Foundation Models and Robotics

We are now looking for a Senior Research Scientist focused on Multimodal Foundation Models and Robotics! NVIDIA is searching for an outstanding research scientist to build humanoid robot foundation models and systems in the Generalist Embodied Agent Research (GEAR) group. Everything that moves will eventually be autonomous. Our mission is to build general-purpose embodied agents that learn to explore and master complex skills across the virtual and the physical world.

You will work with an amazing and collaborative research team that consistently produces influential works on multimodal foundation models, large-scale robot learning, game AI, and physical simulation. Our past projects include Eureka, VIMA, Voyager, MineDojo, MimicPlay, Prismer, and more. One of our team’s most recent milestones includes Project GR00T, a foundation model for humanoid robots. Your contributions will have a significant impact on our moonshot research projects and product roadmaps.

What you will be doing:

• Design and implement novel AI algorithms and models for general-purpose humanoid robots and embodied agents;
• Develop large-scale AI training and inference methods for foundation models;
• Optimize and deploy AI models in physical simulation and on robot hardware;
• Collaborate with research and engineering teams across all of NVIDIA to transfer research to products and services.

What we need to see:

• A Ph.D. in Computer Science/Engineering, Electrical Engineering, etc., or equivalent research experience.
• 5 years of relevant work/research experience across one or both of these fields:
• Multimodal Foundation Models
  • Hands-on training experience and publications in at least one of the following topics: LLMs; Large vision-language models; Video generative models and diffusion algorithms; or Action-based transformers.
  • Outstanding engineering skills in rapid prototyping and model training frameworks (PyTorch, Jax, Tensorflow, etc.). Python is required; C++ and CUDA proficiencies are a big plus;
  • Excellent skills in working with large-scale machine learning/AI systems and compute infrastructure.
• Robotics:
  • Hands-on training experience and publications in robot learning, such as reinforcement learning, imitation learning, classical control methods, etc.
  • Strong programming skills in Python, C++, ROS, and machine learning frameworks like PyTorch.
  • Deep understanding of robot kinematics, dynamics, and sensors;
  • Ability to safely operate robot hardware, lab equipment, and tools;
  • Knowledge of control methods, including PID, model predictive control, and whole-body control;
  • Familiarity with physics simulation frameworks such as MuJoCo and Isaac Sim;
  • Robot hardware design and hands-on building experience.

NVIDIA is widely considered to be one of the technology world's most desirable employers. We have some of the most forward-thinking and productive people in the world. Please join us and be part of the forefront of developing general-purpose robots and embodied agents!

The base salary range is 184,000 USD - 356,500 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

About Us

NVIDIA is the world leader in accelerated computing.

NVIDIA pioneered accelerated computing to tackle challenges no one else can solve. Our work in AI and digital twins is transforming the world's largest industries and profoundly impacting society.