AI Research Evaluator: Scientific Rigor in ML/AI

Lightly AG; United States; Remote; USD 96,000 - 179,000; Part time

Job summary

Lightly AG, a Zurich-based AI company and ETH/HSG spin-off, seeks experienced ML/AI researchers to support an advanced AI research evaluation project. The role focuses on critically assessing AI-generated research against top conference standards and providing evidence-based judgments on scientific validity and novelty.

The project-based, remote engagement offers flexible hours and requires a PhD and a proven publication track record, including multiple NeurIPS/ICML/ICLR papers.

Qualifications

• PhD in ML/AI/CS or related field.
• At least 3 peer-reviewed papers at NeurIPS, ICML, and/or ICLR.
• Experience evaluating AI research and reproducibility.
• Strong analytical and critical reasoning skills.

Responsibilities

• Critically evaluate AI-generated ML/AI research papers, assessing their contributions, technical correctness, methodology, experimental design, and conclusions.
• Assess scientific novelty and significance, determining meaningful advancements over state-of-the-art research.
• Evaluate experimental rigor and validity, including baselines, benchmarks, metrics, statistics, and ablations.
• Identify methodological flaws, unsupported claims, inconsistencies, or misleading results.
• Assess reproducibility and technical soundness, including code and implementation details where available.
• Review relevant literature, identifying overlooked prior work and inaccuracies in citations.
• Evaluate whether AI-generated research meets top-tier conference standards (NeurIPS/ICML/ICLR).
• Provide structured, evidence-based assessments with clear justifications.

Skills

Publications track record

Conference publications

Research evaluation

Education

PhD in ML/AI/CS

Job description

Lightly AG, a Zurich-based AI company and ETH/HSG spin-off, seeks experienced ML/AI researchers to support an advanced AI research evaluation project. The role focuses on critically assessing AI-generated research against top conference standards and providing evidence-based judgments on scientific validity and novelty.

The project-based, remote engagement offers flexible hours and requires a PhD and a proven publication track record, including multiple NeurIPS/ICML/ICLR papers.