AI Research Scientist - Scientific Evaluation (ML/AI)

Lightly

Zürich

Remote

CHF 83,000 - 165,000

Part time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Fully remote
Part-time contractor role
Flexible hours

Job summary

Lightly AG, a Zurich-based AI company, seeks experienced ML/AI researchers to evaluate AI-generated scientific papers for an advanced research evaluation project. This remote, project-based contractor role focuses on scientific validity, novelty, and experimental rigor, measured against top-tier conference standards.

Applicants should hold a PhD in ML/AI/CS or related fields, have published at least three peer-reviewed papers, and show hands-on ML research expertise with strong analytical and

Qualifications

  • PhD or equivalent research expertise.
  • 3+ peer-reviewed papers at NeurIPS/ICML/ICLR (required).
  • Strong hands-on ML/AI research experience.
  • Solid understanding of scientific methodology and experimental rigor.
  • Ability to assess novelty, significance, and technical soundness.
  • Experience reviewing top-tier ML/AI research preferred.
  • Familiarity with state-of-the-art ML/AI research.

Responsibilities

  • Critically evaluate AI-generated ML/AI research papers for scientific validity.
  • Assess novelty, significance, and methodological quality.
  • Evaluate experimental design, baselines, metrics, and robustness.
  • Identify flaws, unsupported claims, or discrepancies.
  • Review reproducibility and available code or details.
  • Survey relevant literature for proper context and citations.
  • Judge whether work meets top-tier conference standards.
  • Provide structured, evidence-based feedback with justification.
  • Apply detailed rubrics and maintain independent judgment.

Skills

Publication track record
Hands-on ML research
Scientific writing
Attention to detail
Independent evaluation

Education

PhD in ML/AI/CS/Statistics or related field

Job description

Lightly AG is a Zurich-based AI company and ETH/HSG spin-off, backed by Y Combinator and top-tier investors. Our machine learning and computer vision technology is trusted by global leaders in autonomous driving, medical imaging, and visual inspection.

We're looking for experienced ML/AI researchers with a strong publication track record at top-tier conferences (NeurIPS, ICML, ICLR) to support an advanced AI research evaluation project. You'll critically assess scientific papers and research outputs generated by autonomous AI research agents, evaluating their technical correctness, scientific novelty, experimental rigor, and overall research quality against the standards of leading ML/AI conferences.

This is a remote, project-based contractor opportunity with flexible working hours.

Tasks
What you'll be doing
  • Critically evaluate AI-generated ML/AI research papers, assessing their scientific contributions, technical correctness, methodology, experimental design, and conclusions
  • Assess scientific novelty and significance, determining whether proposed approaches represent meaningful advancements over existing state-of-the-art research
  • Evaluate experimental rigor and validity, including baseline comparisons, benchmark selection, evaluation metrics, statistical evidence, and ablation studies
  • Identify methodological flaws, unsupported claims, inconsistencies, or misleading results that undermine the scientific validity of proposed contributions
  • Assess reproducibility and technical soundness, including whether reported findings are sufficiently supported by experimental evidence and, where available, accompanying code and implementation details
  • Review and verify relevant academic literature, identifying overlooked prior work, inaccurate citations, and unsupported novelty claims
  • Evaluate whether AI-generated research meets the scientific standards of top-tier ML/AI conferences, such as NeurIPS, ICML, and ICLR
  • Provide structured, evidence-based assessments of research quality, including strengths, weaknesses, critical technical concerns, and overall scientific merit
  • Apply detailed evaluation rubrics consistently, providing clear, well-reasoned justifications for your assessments

The project focuses on evaluating advanced AI-generated scientific research, with an emphasis on technical validity, scientific novelty, experimental rigor, and overall research quality. Your role is to provide expert human judgment on whether the resulting research is scientifically valid, genuinely novel, and of a quality comparable to work accepted at leading ML/AI conferences.

Requirements
  • Have a PhD in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical field, or equivalent demonstrated research expertise
  • Have published at least 3 peer-reviewed research papers at NeurIPS, ICML, and/or ICLR (hard requirement), ideally including first-author or significant research contributions
  • Have substantial hands-on ML/AI research experience, including developing novel methods, designing experiments, analyzing results, and publishing original scientific work
  • Have a strong understanding of scientific methodology and experimental rigor, including appropriate baselines, statistical validity, benchmark design, ablation studies, and reproducibility
  • Can independently assess the novelty, significance, and technical soundness of advanced ML/AI research, identifying both meaningful scientific contributions and fundamental methodological weaknesses
  • Have experience reviewing research papers for top-tier ML/AI conferences or journals, ideally NeurIPS, ICML, or ICLR - strongly preferred
  • Are highly familiar with state-of-the-art ML/AI research and can critically assess new methods against existing literature and established benchmarks
  • Are comfortable verifying scientific claims, including reviewing technical arguments, experimental evidence, citations, and, where relevant, code implementations
  • Have strong analytical and scientific writing skills and can articulate nuanced technical judgments through concise, evidence-based feedback
  • Can consistently apply detailed evaluation guidelines and scoring rubrics while exercising independent expert judgment
  • Have exceptional attention to detail, particularly when identifying subtle methodological flaws, unsupported conclusions, or discrepancies between claims and evidence
Benefits
  • Fully remote and flexible - work from anywhere
  • Part-time contractor role with flexible hours
  • Work on the evaluation of cutting-edge autonomous AI systems capable of conducting end-to-end scientific research
  • Apply your research expertise to help assess and advance the scientific capabilities of frontier AI agents

We look forward to hearing from you!

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML Research Evaluator — Remote Contract
Senior ML Research Evaluator — Remote Contract

Lightly • Zürich

Remote
CHF 83,000 - 165,000
Fully remote
Part-time contractor role
Flexible hours
AI Research Engineer (Agentic Reinforcement Learning)
AI Research Engineer (Agentic Reinforcement Learning)

Jobgether • Switzerland

Remote
CHF 120,000 - 180,000
Fully Remote
International Collaboration
Cutting-Edge AI Research
+3
Senior FP&A Expert - Financial Task Design and Evaluation
Senior FP&A Expert - Financial Task Design and Evaluation

Lightly • Switzerland

Remote
CHF 76,000 - 91,000
Fully remote
Flexible hours
Part-time contractor
Senior FP&A Expert - Financial Task Design and Evaluation
Senior FP&A Expert - Financial Task Design and Evaluation

Lightly AG • Zürich

Remote
CHF 375,000 - 534,000
Fully remote
Flexible hours
Part-time contractor
Senior FP&A Expert - Financial Task Design and Evaluation
Senior FP&A Expert - Financial Task Design and Evaluation

Lightly • Zürich

Remote
CHF 382,000 - 543,000
Fully remote
Flexible hours
Contractor role
AI Scientist
AI Scientist

Artificialy • Zürich

On-site
CHF 120,000 - 180,000
AI Research Engineer (Kernel & Inference Optimization)
AI Research Engineer (Kernel & Inference Optimization)

Jobgether • Switzerland

Remote
EUR 137,000 - 222,000
Remote-first team
International team
Cutting-edge AI research
+3
Freelance Web & CloudOps Experts at AI Startup
Freelance Web & CloudOps Experts at AI Startup

Lightly • Zürich

Hybrid
CHF 83,000 - 138,000
YC-backed team
Competitive compensation tailored to経験
Applied AI Engineer
Applied AI Engineer

Jobgether • Switzerland

Remote
CHF 92,000 - 138,000
Remote work across Europe/UK
Early core team influence
Technical autonomy
+4
AI/ML Engineer
AI/ML Engineer

Jobgether • Switzerland

Remote
CHF 140,000 - 190,000
Remote work
Flexible schedule
International collaboration
Top locations