Research Scientist Intern, Audio-Visual Conversation Scene Generation (PhD)

Meta; Redmond; On-site; AUD 72,000 - 116,000; Full time

Job summary

Meta is seeking a Reality Labs Research Audio intern to help generate realistic video with audio of multi-person conversations. The role involves controlling 3D scene positions, gaze and turn-taking within egocentric camera setups to create accurately labelled data for audio-visual models.

The candidate should be pursuing a PhD in CS/EE/ML or related fields and have strong research experience in generative video/image modeling, with publications at major venues and hands-on diffusion or

Qualifications

• Currently has or is pursuing a PhD in Computer Science, Electrical Engineering, Machine Learning or a related field.
• Research experience in generative modeling for video or images with first-author publications at CVPR, ICCV, ECCV, NeurIPS, ICLR or SIGGRAPH.
• Hands-on experience training or fine-tuning modern video diffusion or flow-matching models (e.g. DiT-based architectures).
• Experience with controllable generation from geometric signals: camera pose, 3D layout, depth or human pose.
• Solid grounding in 3D vision: camera models, multi-view geometry and coordinate transforms.
• 3+ years experience with Python, Matlab, PyTorch, large-scale distributed GPU training or similar.
• Experience with joint audio-video generation, speech-driven talking heads or gestures, multi-speaker TTS, or lip sync.
• Background in spatial audio: binaural rendering, HRTFs, or room acoustics simulation.
• Research on modeling social interaction: conversational dynamics, gaze, turn-taking or group formations.
• Familiarity with egocentric perception and datasets (Ego4D, Ego-Exo4D, Aria) and related tasks.
• Experience with 3D or 4D human and scene representations (SMPL-X, Gaussian splatting, NeRF) or multi-view-consistent video generation.
• Experience with synthetic data generation, game engines, or improving model performance on real data.
• Experience using LLMs or VLMs to plan scenes, layouts or dialogue.
• Track record of open-source contributions or a previous industry research internship.
• Proven track record of results shown in accepted papers at top conferences like CVPR, ECCV, NeurIPS, ICASSP, InterSpeech.
• Experience working and communicating cross-functionally in a team.
• Intent to return to a degree-program after completion of internship/co-op.

Responsibilities

• Generate realistic video with audio of multi-person conversations with controllable 3D positioning.
• Develop scenes with natural social interactions: gaze, turn-taking and group formations.
• Prepare labelled training data for audio-visual models using egocentric camera setups.
• Collaborate with researchers, engineers and product teams on project goals.

Skills

Python

Matlab

PyTorch

Large-scale distributed GPU training

3D/4D vision concepts

EGocentric perception basics

Education

PhD in Computer Science/EE/ML or related field

Tools

Python

Matlab

PyTorch

Job description

The Meta Reality Labs Research Team brings together a world-class team of researchers, developers, and engineers to create the future of virtual and augmented reality, which together will become as universal and essential as smartphones and personal computers are today.

Reality Labs Research Audio builds the audio-visual perception that future smart glasses will rely on. That includes understanding who the wearer is talking to, who is speaking, and how a conversation is organized. To train and evaluate these models at scale, we need realistic conversation data that comes with exact ground truth. This internship explores generative models that produce such data.

The project:

• Generate realistic video with audio of multi-person conversations. Inputs can be text alone, text plus one image of a group, or text plus a reference image for each person. Beyond looking real, the scenes should show natural social interaction: turn-taking, gaze, head orientation, listener reactions and group formations. Each scene should also be controllable: the 3D position of each person, how people are grouped, and the position and pose of the egocentric camera. That control is what lets the output serve as accurately labelled training data for audio-visual models, e.g. predicting the glasses wearer's intent or detecting conversation groups.
• Currently has, or is in the process of obtaining, a PhD in Computer Science, Electrical Engineering, Machine Learning or a related field
• Research experience in generative modeling for video or images, with first-author publications at venues such as CVPR, ICCV, ECCV, NeurIPS, ICLR or SIGGRAPH
• Hands-on experience training or fine-tuning modern video diffusion or flow-matching models (e.g. DiT-based architectures), including conditioning methods such as adapters, ControlNet or LoRA
• Experience with controllable generation from geometric or structural signals: camera pose, 3D layout, depth or human pose
• Solid grounding in 3D vision: camera models, multi-view geometry and coordinate transforms
• 3+ years experience with Python, Matlab, Pytorch, large-scale distributed GPU training or similar
• Must obtain work authorization in the country of employment at the time of hire, and maintain ongoing work authorization during employment
• Experience with joint audio-video generation, speech-driven talking heads or gestures, multi-speaker TTS, or lip sync
• Background in spatial audio: binaural rendering, HRTFs, or room acoustics simulation
• Research on modeling social interaction: conversational dynamics, gaze, turn-taking or group formations
• Familiarity with egocentric perception and datasets (e.g. Ego4D, Ego-Exo4D, Aria), and with tasks such as active speaker detection or conversation grouping
• Experience with 3D or 4D human and scene representations (e.g. SMPL-X, Gaussian splatting, NeRF) or multi-view-consistent video generation
• Experience with synthetic data generation, game engines, or improving how well models trained on synthetic data perform on real data
• Experience using LLMs or VLMs to plan scenes, layouts or dialogue
• Track record of open-source contributions or a previous industry research internship
• Proven track record of achieving results as demonstrated in accepted papers at top computer vision and machine learning related conferences such as CVPR, ECCV, NIPS, ICASSP, InterSpeech etc
• Experience working and communicating cross functionally in a team environment
• Intent to return to a degree-program after the completion of the internship/co-op