AI Research Scientist: Fairness, LLM Benchmarking & Scoring

MakiPeople

Paris (TX)

On-site

USD 120,000 - 180,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Maki People is seeking an AI Research Scientist to advance scientifically robust, fair automated scoring systems. You will contribute to evaluating LLMs, building self-hosted scoring models, and calibrating psychometric methods while collaborating with engineers and product teams.

You will design studies comparing AI outputs with expert judgments, publish findings, and help shape governance and transparency around our AI solutions.

Qualifications

  • PhD or MSc in ML, linguistics, data science or psychology.
  • Publication track record (peer‑reviewed papers or conference papers).
  • Hands-on experience evaluating LLMs: benchmarking, measuring agreement with human raters, testing robustness.
  • Experience testing AI-driven decisions for fairness and bias.
  • Proficiency in Python or R, with statistical analysis.
  • Clear communication of results to scientific and business audiences.

Responsibilities

  • Design studies to compare AI scores with expert ratings and human benchmarks.
  • Develop hybrid scoring pipelines combining AI with human oversight.
  • Calibrate psychometric models for validity across populations and forms.
  • Analyze bias across demographic groups and apply mitigation techniques.
  • Publish reports and present findings to both technical and non-technical stakeholders.

Skills

Python
R
Statistical analysis
LLM evaluation
Communication
LLM benchmarking

Education

PhD or MSc in machine learning, computational linguistics, data science or psychology

Tools

LoRA fine-tuning
Cloud databases

Job description

Maki People is seeking an AI Research Scientist to advance scientifically robust, fair automated scoring systems. You will contribute to evaluating LLMs, building self-hosted scoring models, and calibrating psychometric methods while collaborating with engineers and product teams.

You will design studies comparing AI outputs with expert judgments, publish findings, and help shape governance and transparency around our AI solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Applied Scientist: Client Scoring & Fairness
AI Applied Scientist: Client Scoring & Fairness

MakiPeople • Paris (TX)

On-site
USD 120,000 - 180,000
AI & Law Research Scientist — Benchmarks & Evaluation
AI & Law Research Scientist — Benchmarks & Evaluation

Connect Legal Partners • San Francisco (CA)

On-site
USD 185,000 - 250,000
Equity
ML Evaluation Scientist - LLM Benchmarks
ML Evaluation Scientist - LLM Benchmarks

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health coverage
Retirement benefits
L&D stipend
+2
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Scale AI, Inc. • San Francisco (CA)

On-site
USD 166,000 - 207,000
Health coverage
Equity
Retirement benefits
+3
AI Fairness & Bias Data Scientist, People Analytics
AI Fairness & Bias Data Scientist, People Analytics

AIToolboard • United States

On-site
USD 180,000 - 240,000
LLM Evaluation Scientist — Benchmarks & Failure Analysis
LLM Evaluation Scientist — Benchmarks & Failure Analysis

Scale AI, Inc. • Seattle (WA)

On-site
USD 181,000 - 226,000
Health, dental and vision coverage
Retirement benefits
Learning and development stipend
+3
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc. • New York (NY)

On-site
USD 181,000 - 226,000
Base salary + equity
Health, dental, vision coverage
Retirement benefits
+3
Senior AI Model Evaluation Scientist (LLM Benchmarks)
Senior AI Model Evaluation Scientist (LLM Benchmarks)

Cohere • Seattle (WA)

Remote
USD 180,000 - 385,000
Lunch stipend
Health and dental benefits
RRSP matching / 401K / Pension
+5
Senior AI Researcher — Equity & High-Impact Legal AI
Senior AI Researcher — Equity & High-Impact Legal AI

Ivo • San Francisco (CA)

On-site
USD 200,000 - 325,000
Competitive Compensation
Equity
Relocation and Visa Support
+6
Client-Facing AI Scientist: Scoring & Insights
Client-Facing AI Scientist: Scoring & Insights

Maki • Paris (TX)

On-site
USD 120,000 - 180,000