AI Research Scientist: LLM Benchmarking & Fairness

Maki

Paris

On-site

EUR 70,000 - 110,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Maki is seeking an AI Research Scientist to advance the science and AI behind our platform. You will work at the intersection of AI, psychometrics and research to ensure our automated scoring is scientifically robust and fair.

Responsibilities include evaluating LLMs, benchmarking across languages, developing hybrid AI-human scoring, and publishing findings. A PhD or MSc and strong Python or R skills are required.

Qualifications

  • PhD or MSc in machine learning, computational linguistics, data science or psychology.
  • Hands-on experience evaluating LLMs: benchmarks, agreement with human raters, robustness.
  • Experience testing AI-driven decisions for fairness and bias; clear results communication.

Responsibilities

  • Evaluate and test LLMs for scoring accuracy and reliability across languages.
  • Benchmark models and prompts; explore prompts and model optimisation for production.
  • Build and calibrate hybrid AI-human scoring pipelines with engineering teams.
  • Design studies to compare AI scores with expert ratings and ensure validity.
  • Publish reports, papers or conference materials and explain findings clearly.

Skills

LLM evaluation
Benchmarking
Prompt engineering
Model robustness
Python or R
Communication
Fairness and bias testing

Education

PhD or MSc in ML/Computational linguistics/data science/psychology

Tools

Python
R
Cloud databases

Job description

Maki is seeking an AI Research Scientist to advance the science and AI behind our platform. You will work at the intersection of AI, psychometrics and research to ensure our automated scoring is scientifically robust and fair.

Responsibilities include evaluating LLMs, benchmarking across languages, developing hybrid AI-human scoring, and publishing findings. A PhD or MSc and strong Python or R skills are required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Research Scientist: LLM Benchmarks & Fairness
AI Research Scientist: LLM Benchmarks & Fairness

MakiPeople • Paris

On-site
EUR 90,000 - 130,000
AI Applied Scientist: Elevate Hiring, Scoring & Fairness
AI Applied Scientist: Elevate Hiring, Scoring & Fairness

MakiPeople • Paris

On-site
EUR 65,000 - 110,000
AI Applied Scientist
AI Applied Scientist

MakiPeople • France

Hybrid
EUR 90,000 - 130,000
AI Solutions Scientist — Client Impact & Fair Scoring
AI Solutions Scientist — Client Impact & Fair Scoring

MakiPeople • France

Hybrid
EUR 90,000 - 130,000
AI Research Scientist
AI Research Scientist

MakiPeople • Paris

On-site
EUR 90,000 - 130,000
Applied AI Evaluation Engineer: LLM Benchmarks & Pipelines
Applied AI Evaluation Engineer: LLM Benchmarks & Pipelines

Mistral • Paris

On-site
EUR 90,000 - 130,000
Healthcare coverage
Parental leave
Retirement plans
+3
AI Research Scientist
AI Research Scientist

Maki • Paris

On-site
EUR 70,000 - 110,000
Remote LLM Evaluation Scientist - Benchmark Frontiers
Remote LLM Evaluation Scientist - Benchmark Frontiers

Anyone AI • Paris

Remote
EUR 90,000 - 140,000
AI Applied Scientist: Client Impact & Fairness Champion
AI Applied Scientist: Client Impact & Fairness Champion

MakiPeople • Paris

On-site
EUR 70,000 - 110,000
ML Benchmark Researcher
ML Benchmark Researcher

IXO Inc. • France

Remote
EUR 166,000 - 222,000