Research Scientist Intern - Multimodal Sensing & On-Device Perception - Global Frontier Tech Recruitment Program - 2027 Start (PHD)

ByteDance; San Jose (CA); On-site; USD 85,000 - 121,000; Full time

Benefits offered by this job

Health insurance

Life insurance

Wellbeing benefits

Housing allowance

Job summary

ByteDance's Pico unit is seeking PhD interns for 2027 to contribute to eye-tracking hardware research and AI perception improvements. You will work on sensor front-end architectures, end-to-end imaging pipelines, and co-design of hardware and vision models.

The internship blends hands-on learning with collaboration across optics, sensors, and CV/ML teams, offering exposure to real-world research and potential product impact.

Qualifications

• Currently pursuing a PhD in a technical field (CS/EE/Optical/Math/Physics).
• Strong research background in computer vision and ML with hands-on model training.
• Experience with sequence modeling, language modeling, efficient neural networks, or signal processing.

Responsibilities

• Design and prototype sensor/imaging architectures with near-sensor processing.
• Build end-to-end imaging pipelines from optical physics to perception models.
• Leverage VLM/LLM knowledge to guide sensing front-end requirements.
• Develop or adapt vision models co-optimized with hardware constraints.

Skills

PhD in relevant field

CV/ML research

Sequence modeling

Language modeling

Efficient neural networks

Signal processing

Education

PhD in CS/EE/Optical Engineering/Applied Math/Physics or related field

Tools

Hardware prototyping

Embedded ML deployment

Job description

Responsibilities

Team Introduction: The eye tracking system architecture team works on the vertical stack of eye tracking systems architecture and related key components. Our goal is enabling an eye tracking system with high population coverage, high performance, and low power consumption. Eye tracking is a highly multidiscipline area. You will have the opportunity to work on state of art technologies across refractive/diffractive optics, image sensor, human factor and CV algorithm.

Team Introduction: The eye tracking system architecture team works on the vertical stack of eye tracking systems architecture and related key components. Our goal is enabling an eye tracking system with high population coverage, high performance, and low power consumption. Eye tracking is a highly multidiscipline area. You will have the opportunity to work on state of art technologies across refractive/diffractive optics, image sensor, human factor and CV algorithm.

Responsibilities

We are looking for talented individuals to join us for an internship in 2027. PhD Internships at our Company aim to provide students with the opportunity to actively contribute to our products and research, and to the organization's future plans and emerging technologies.

PhD internships at Our Company provides students with the opportunity to actively contribute to our products and research, and to the organization's future plans and emerging technologies. Our dynamic internship experience blends hands‑on learning, enriching community‑building and development events, and collaboration with industry experts.

Topic Content

The rapid advancement of AI Agents is set to have a far-reaching and transformative impact on daily life. As the hardware bridge linking AI Agents to the physical world and end users, intelligent hardware acts as a vital gateway for agents to perceive their environment and recognize user intent. Enabling AI Agents to maintain round‑the‑clock environmental awareness and efficiently capture real‑time user intentions is essential to improving their everyday service experience. This topic seeks to overcome the limitations of conventional visual perception systems. By deeply integrating sensing and computing, it explores full‑stack innovation spanning from low‑level hardware to high‑level algorithms. Key efforts include developing next‑generation sensors with real‑time understanding capabilities, investigating non‑traditional visual capture and compression techniques, and designing novel hardware architectures to run advanced algorithms—enabling highly efficient coordination across sensing, processing, and communication.

Topic Challenge

Break free from the framework of conventional sensing systems, explore novel sensors, signal processing and compression schemes to achieve highly energy‑efficient sensing tasks, while enabling seamless integration with large models.

Topic Value

Breakthroughs in this research direction will enable intelligent hardware to better connect AI, users, and daily life. Against the backdrop of the AI Agent era, this will unlock the gateway to the next generation of intelligent hardware terminals and open up broader technological possibilities.

Project Objective

Rethink visual perception from the sensor up. Instead of treating sensing and computation as separate stages, we're exploring architectures that fuse them — sensors that compress, select, and partially "understand" the scene before data ever leaves the chip. This requires deeply understanding both the hardware constraints and what modern vision‑language and world models actually need from the sensing front‑end.

The goal: order‑of‑magnitude gains in perception efficiency by co‑designing hardware and algorithms across the full pipeline.

Responsibilities

• Design and prototype novel sensor or imaging architectures that move computation closer to the sensing front‑end (e.g., near‑sensor processing, event‑driven capture, learned compression at the pixel level)
• Build and characterize imaging pipelines end‑to‑end: from optical/sensor physics through ISP to downstream perception models — identify where bits are wasted and where intelligence should be injected
• Leverage understanding of VLM/LLM and world models to inform what information the sensing front‑end must preserve, discard, or transform — closing the loop between foundation model requirements and hardware design
• Develop or adapt machine vision models that are co‑optimized with hardware constraints (power, bandwidth, latency)

Minimum Qualifications

• Currently pursuing a PhD in Computer Science, Electrical Engineering, Optical Engineering, Applied Mathematics, Physics, or a related technical field.
• Strong research background in computer vision and machine learning, with hands‑on model training experience.
• Experience with at least one of: sequence modeling, language modeling, efficient neural network design, or signal processing.

Preferred Qualifications

• Proven track record of high‑impact research, demonstrated by publications in CVPR, ICCV, ECCV, NeurIPS, ICLR, SIGGRAPH, or similar.
• Hands‑on experience with hardware prototyping involving cameras, structured light, or other active sensing systems.
• Familiarity with on‑device or on‑sensor compute, embedded ML deployment, and hardware/software co‑design tradeoffs.
• Self‑motivated, curious, and excited about taking ambitious research ideas from prototype to working system.

Job Information

【For Pay Transparency】Compensation Description (Hourly) - Campus Intern

The hourly rate range for this position in the selected city is $75- $75.

Benefits may vary depending on the nature of employment and the country work location. Interns have day one access to health insurance, life insurance, wellbeing benefits and more. Interns also receive 10 paid holidays per year and paid sick time (56 hours if hired in first half of year, 40 if hired in second half of year). Interns who are not working 100% remote may also be eligible for housing allowance.

The Company reserves the right to modify or change these benefits programs at any time, with or without notice.

For Los Angeles County (unincorporated) Candidates:

• Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;
• Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems; and
• Exercising sound judgment.

About Pico

Founded in April 2015, Pico is a VR company committed to developing immersive and interactive VR experiences for people around the world. Pico also provides tailor‑made solutions for our enterprise clients in the fields of education and healthcare.

Why Join ByteDance

Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.

As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Reasonable Accommodation

ByteDance is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/RA-request