AI Research Scientist – Scientific Evaluation (ML/AI)

Lightly AG; United States; A distancia; USD 96.000 - 179.000; A tiempo parcial

Descripción de la vacante

Lightly AG, a Zurich-based AI company and ETH/HSG spin-off, seeks experienced ML/AI researchers to support an advanced AI research evaluation project. The role focuses on critically assessing AI-generated research against top conference standards and providing evidence-based judgments on scientific validity and novelty.

The project-based, remote engagement offers flexible hours and requires a PhD and a proven publication track record, including multiple NeurIPS/ICML/ICLR papers.

Formación

• PhD in ML/AI/CS or related field.
• At least 3 peer-reviewed papers at NeurIPS, ICML, and/or ICLR.
• Experience evaluating AI research and reproducibility.
• Strong analytical and critical reasoning skills.

Responsabilidades

• Critically evaluate AI-generated ML/AI research papers, assessing their contributions, technical correctness, methodology, experimental design, and conclusions.
• Assess scientific novelty and significance, determining meaningful advancements over state-of-the-art research.
• Evaluate experimental rigor and validity, including baselines, benchmarks, metrics, statistics, and ablations.
• Identify methodological flaws, unsupported claims, inconsistencies, or misleading results.
• Assess reproducibility and technical soundness, including code and implementation details where available.
• Review relevant literature, identifying overlooked prior work and inaccuracies in citations.
• Evaluate whether AI-generated research meets top-tier conference standards (NeurIPS/ICML/ICLR).
• Provide structured, evidence-based assessments with clear justifications.

Conocimientos

Publications track record

Conference publications

Research evaluation

Educación

PhD in ML/AI/CS

Descripción del empleo

Lightly AG is a Zurich-based AI company and ETH/HSG spin-off, backed by Y Combinator and top-tier investors. Our machine learning and computer vision technology is trusted by global leaders in autonomous driving, medical imaging, and visual inspection. We’re looking for experienced ML/AI researchers with a strong publication track record at top-tier conferences (NeurIPS, ICML, ICLR) to support an advanced AI research evaluation project. You'll critically assess scientific papers and research outputs generated by autonomous AI research agents, evaluating their technical correctness, scientific novelty, experimental rigor, and overall research quality against the standards of leading ML/AI conferences. This is a remote, project-based contractor opportunity with flexible working hours.

Tasks

What you’ll be doing

• Critically evaluate AI-generated ML/AI research papers, assessing their scientific contributions, technical correctness, methodology, experimental design, and conclusions
• Assess scientific novelty and significance, determining whether proposed approaches represent meaningful advancements over existing state-of-the-art research
• Evaluate experimental rigor and validity, including baseline comparisons, benchmark selection, evaluation metrics, statistical evidence, and ablation studies
• Identify methodological flaws, unsupported claims, inconsistencies, or misleading results that undermine the scientific validity of proposed contributions
• Assess reproducibility and technical soundness, including whether reported findings are sufficiently supported by experimental evidence and, where available, accompanying code and implementation details
• Review and verify relevant academic literature, identifying overlooked prior work, inaccurate citations, and unsupported novelty claims
• Evaluate whether AI-generated research meets the scientific standards of top-tier ML/AI conferences, such as NeurIPS, ICML, and ICLR
• Provide structured, evidence-based assessments of research quality, including strengths, weaknesses, critical technical concerns, and overall scientific merit
• Apply detailed evaluation rubrics consistently, providing clear, well-reasoned justifications for your assessments

The project focuses on evaluating advanced AI-generated scientific research, with an emphasis on technical validity, scientific novelty, experimental rigor, and overall research quality. Your role is to provide expert human judgment on whether the resulting research is scientifically valid, genuinely novel, and of a quality comparable to work accepted at leading ML/AI conferences.

Requirements

You're a strong candidate if you:

• Have a PhD in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical field, or equivalent demonstrated research expertise
• Have published at least 3 peer-reviewed research papers at NeurIPS, ICML, and/or ICLR (hard requirement), ideally including first-author or significant research contributions
• Have substant