Stage - Training networks to be compressible: low rank, quantization, and how they combine-Saclay-H/F

CEA; Saclay; In loco; EUR 12.000 - 17.000; Part-time

Descrizione del lavoro

Le LIST du CEA à Saclay recherche un stagiaire en ML pour étudier la compressibilité des réseaux neuronaux par faible rang et quantification, et explorer l’impact des préparations en formation sur les coûts de quantification.

Le candidat contribuera à des expériences avec des modèles linguistiques et à la rédaction de résultats publiables, encadré par une équipe de recherche spécialisée.

Competenze

• Master 2 ou équivalent en ML/CS (stage étudiants).
• Compétences fortes en algèbre linéaire et optimisation.

Mansioni

• Mesurer l’impact de l’entraînement vers le faible rang sur la compressibilité d’un modèle.
• Réaliser des expériences avec factorisation et quantification à mémoire égale.
• Proposer et tester une amélioration basée sur le prix calculé par l’équipe.

Conoscenze

Python

PyTorch

Hugging Face

Algèbre linéaire

Optimisation

Conception d’expériences

Formazione

Master 2 ou école d’ingénieurs en ML/CS

Descrizione del lavoro

Stage - Training networks to be compressible: low rank, quantization, and how they combine-Saclay-H/F

Contrat

Stage

Stage - Training networks to be compressible: low rank, quantization, and how they combine-Saclay-H/F

To run on smaller hardware or serve more cheaply, large neural networks are often compressed before deployment: most commonly by quantization (storing numbers with few bits), less often by low-rank factorization (replacing a weight matrix by a product of two thin factors). Models are increasingly trained or fine-tuned so that they compress well, for instance by pushing their weights toward low rank. These methods are mostly judged on factorization alone, yet a factorized model is also quantized before it is deployed. Does preparing a model for low rank help or hurt once it is also quantized, and can the combination compete with quantization alone? Our first measurements suggest the answer is less simple than it looks. Our group develops compression methods that measure errors by their effect on the network's outputs, and that predict in closed-form what rounding low-rank factors costs.

As an intern at the CEA, you will have the opportunity to work in a world-renowned research environment. Our teams consist of passionate and dedicated experts, providing an environment conducive to learning and collaboration. You will have access to state-of-the-art equipment and top-tier research resources to carry out your assignments. The work performed may potentially lead to a scientific publication.

Context:

This internship aims to understand how training a network toward low rank changes what it costs to quantize its factors, using the team's closed-form price of rounding both to explain the effect and to act on it. It combines matrix analysis with controlled experiments on language models, to:

What do we expect from you?

• Measure what low-rank preparation does to a model's compressibility, by factorization and by quantization, against an unprepared model and against quantization alone, at equal memory.
• Explain it: what the preparation changes in the factors, read through the closed-form price, and whether the price predicts the measured cost.
• Use the price to propose and test an improvement.

The internship may lead to a PhD starting in October 2027.

#Cea List

Profile:

• Master's (M2) or engineering-school student in machine learning, applied mathematics or computer science
• Strong linear algebra; probability and optimization are a plus
• Solid Python and PyTorch
• Care in designing experiments and reading their results
• Experience with Hugging Face language models is a plus

Site

Saclay

Saclay

Langues

Oui

01/02/2027

Référence

2026-41976

Based in Saclay (Essonne), the LIST is one of the two institutes of CEA Tech, the Technological Research Division of the CEA. Dedicated to intelligent digital systems, its mission is to carry out technological developments of excellence on behalf of industrial partners, in order to create value.
Within the LIST, the Laboratory of Vision for Modeling and Localization (LVML) conducts its research in the field of computer vision and artificial intelligence for the perception of intelligent and autonomous systems. The laboratory's research themes include 3D localization, segmentation, characterization and vision for robotics.