Post-Doctoral Research Visit F/M Stochastic non-smooth optimization by first-order methods

1000scholars; Saint-Martin-d'Hères; On-site; EUR 36,000 - 54,000; Full time

Job summary

1000scholars seeks a postdoctoral researcher to deepen the understanding of adaptive optimization algorithms that adjust hyperparameters based on accumulated data. You will work with Inria GHOST and LJK teams on AdaGrad variants in stochastic, non-convex, and non-smooth contexts.

The role focuses on defining problem classes near learning models and analyzing nonsmooth algorithms, with dissemination of results through articles and conferences.

Qualifications

• Foundation knowledge in numerical optimization, in particular first-order methods.
• Scientific programming skills (Julia, Python) for empirical validations of algorithms.

Responsibilities

• Identify necessary/sufficient conditions so adaptive algorithms converge.
• Investigate robustness under noise, initialization, and initial step-size.
• Compare properties with classical methods like stochastic gradient descent.
• Map a zoology of structured optimization problems where adaptive methods outperform.

Skills

Adaptive optimization

AdaGrad/Adam/RMSProp

Scientific programming

Python

Julia

Education

PhD in optimization / ML

Tools

Python

Julia

Job description

Context

The postdoc will be conducted between the Inria GHOST team and the DAO team of the LJK lab. It will be funded by the Structured Optimization and Learning chair of the MIAI cluster.

The main objective is to analyze and deepen our understanding of adaptive optimization algorithms which adapt their hyperparameter (e.g., the step size) based on the data accumulated during the optimization process (e.g., during training in a learning context). This family includes central algorithms such as AdaGrad, RMSProp, Adam, and related variants, which have a key role in the recent development of numerical optimization. The majority of existing work considers these algorithms applied on smooth functions, and we want to extend this understanding to stochastic, non-convex, and non-smooth optimization. Those results will contribute to narrow the gap between current theory and the practical applications in which these algorithms are already used, which are non-convex, non-smooth.

Assignment

The recruited candidate will work by Mathieu Besançon (Inria, LIG), Quoc-Tung Le (UGA, LJK) and Jérôme Malick (CNRS, LJK).

To understand the topic:

Some important algorithms studied in the smooth case are currently ill-understood when applied to nonsmooth instances. One of the research questions will focus on the definition of a problem class close to learning models and on which nonsmooth algorithms can be analyzed.

The postdoctoral researcher is expected to conduct research on AdaGrad algorithms in the nonsmooth setting.

Collaboration

Some collaborations with the Zuse Institute Berlin and the Institut Mathématique de Toulouse will be possible during the working period.

Main activities

Principal activities:

• Identify necessary/sufficient conditions so that adaptive algorithms can converge.
• Investigate the robustness of the optimization methods, i.e., study qualitative and quantitative properties under the effects of noises, initialization, initial step-size.
• Comparing these properties of adaptive algorithms with those of classical ones, such as (stochastic) gradient descent.
• Identify a zoology of structured optimization problems, where adaptive algorithms can significantly outperform other families of methods.

Complementary activities:

• Disseminating findings via articles, seminars and conferences.

Skills

Foundation knowledge in numerical optimization, in particular on first-order methods.

Scientific programming skills (Julia, Python) for empirical validations of algorithms.