Lead AI Data Quality & Evaluation Engineer

IllTip

United States

On-site

USD 180,000 - 280,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

IllTip is seeking a leadership research engineering head to drive a group focused on measuring and improving data quality for AI training and evaluation. You will oversee reinforcement learning environments, synthetic datasets, benchmarks, and domain-specific workflows, setting the quality strategy and the systems behind it.

You will lead a team of researchers and engineers, turning research findings into production tools, dashboards, and validation pipelines, while mentoring staff and

Qualifications

  • Minimum 5 years of research or engineering experience in AI data evaluation and quality systems.
  • Experience leading technical teams or projects from problem definition through implementation.
  • Advanced proficiency in Python, Docker and Linux environments.
  • Strong background in AI evaluations, post-training, synthetic data validation, benchmarks or related evaluation infrastructure.
  • Clear written communication and startup environment experience.

Responsibilities

  • Lead a team that builds systems to evaluate large volumes of tasks across environments, datasets, benchmarks and specialized workflows.
  • Define data quality strategy by building quality-control systems, setting standards and designing experiments to assess agent outputs.
  • Develop scalable methods to validate synthetic data, including failure-mode analysis, task mutation checks and trajectory auditing.
  • Work with research engineers, domain experts and data vendors to diagnose quality problems and improve data generation.
  • Turn research findings into production tools, dashboards, validation pipelines and feedback loops.
  • Set rigorous standards for judging whether agent training data is useful, and mentor research engineers.

Skills

Python
Clear written communication
Startup experience

Tools

Docker
Linux

Job description

IllTip is seeking a leadership research engineering head to drive a group focused on measuring and improving data quality for AI training and evaluation. You will oversee reinforcement learning environments, synthetic datasets, benchmarks, and domain-specific workflows, setting the quality strategy and the systems behind it.

You will lead a team of researchers and engineers, turning research findings into production tools, dashboards, and validation pipelines, while mentoring staff and

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding AI Evaluation Engineer
Founding AI Evaluation Engineer

IllTip • San Francisco (CA)

On-site
USD 180,000 - 280,000
AI Data Quality Engineer for Post-Training & Evaluation
AI Data Quality Engineer for Post-Training & Evaluation

Reflection AI • San Francisco (CA)

On-site
USD 190,000 - 230,000
Stock options
Health & wellness
Meals provided in office
+1
Lead Research Engineer, Data Quality (Visa Sponsorship - United States)
Lead Research Engineer, Data Quality (Visa Sponsorship - United States)

IllTip • United States

On-site
USD 180,000 - 280,000
Remote AI QA Architect & Evaluation Lead
Remote AI QA Architect & Evaluation Lead

Meridial • United States

Remote
USD 8,300 - 90,000
In-Person SF-Based Research Engineer: Scalable AI Data Systems
In-Person SF-Based Research Engineer: Scalable AI Data Systems

Idler • San Francisco (CA)

On-site
USD 140,000 - 210,000
Competitive salary
Meaningful equity
Free meals
+5
Lead, AI Quality & Reliability Engineering
Lead, AI Quality & Reliability Engineering

Vizient • Edina (MN)

On-site
USD 88,900 - 155,500
Senior AI Evaluation & Benchmarking Lead
Senior AI Evaluation & Benchmarking Lead

TRSS • Ann Arbor (MI)

Hybrid
USD 127,000 - 237,000
Hybrid Work Model
Mental Health Days
Headspace app
+2
Staff Data Scientist, AI Evaluation
Staff Data Scientist, AI Evaluation

Syllo • United States

Remote
USD 120,000 - 165,000
Senior AI Quality & Data Strategy Lead
Senior AI Quality & Data Strategy Lead

HERE • United States

Remote
USD 160,000 - 170,000
Health insurance
Retirement savings plans
Paid time off
Lead AI Evaluation & Benchmarking Engineer (Hybrid)
Lead AI Evaluation & Benchmarking Engineer (Hybrid)

Thomson Reuters  • New York (NY)

Hybrid
USD 127,000 - 237,000
Hybrid Work Model
Tuition Reimbursement
Mental Health Days