Founding AI Evaluation Engineer

IllTip

San Francisco (CA)

On-site

USD 180,000 - 280,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

IllTip is seeking a founding engineer in San Francisco to design and build evaluation environments and benchmarks for AI agents, ensuring agent performance reaches real-world production readiness. This is a founding role with a research-forward, hands-on engineering focus in a fast-growing AI startup.

You will create synthetic company environments, build graders, and develop benchmarks that align with practical production outcomes, while addressing challenges like reward hacking and benchmark

Qualifications

  • Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
  • Experience building environments or benchmarks used to train or evaluate agents, and designing graders or evaluation systems for AI agents.
  • Experience simulating complex software systems for agent tool use, and finding benchmark flaws or fixing reward hacking.

Responsibilities

  • Design synthetic company environments with realistic email, records, and files that support extended agent use.
  • Build robust graders and evaluation systems that can judge agent output even where human reviewers are not expert enough to do so.
  • Develop benchmarks that correlate with real-world production performance.
  • Identify and fix evaluation failures, including reward hacking and benchmark gaming.
  • Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.

Skills

Synthetic data pipelines
Benchmarks & evaluations
Agent tool environments
Research & engineering

Job description

IllTip is seeking a founding engineer in San Francisco to design and build evaluation environments and benchmarks for AI agents, ensuring agent performance reaches real-world production readiness. This is a founding role with a research-forward, hands-on engineering focus in a fast-growing AI startup.

You will create synthetic company environments, build graders, and develop benchmarks that align with practical production outcomes, while addressing challenges like reward hacking and benchmark

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding Engineer — Autonomous AI Agents
Founding Engineer — Autonomous AI Agents

IllTip • San Francisco (CA)

On-site
USD 150,000 - 230,000
Lead AI Data Quality & Evaluation Engineer
Lead AI Data Quality & Evaluation Engineer

IllTip • United States

On-site
USD 180,000 - 280,000
Founding Engineer, Environments and Data (Visa Sponsorship - San Francisco, United States)
Founding Engineer, Environments and Data (Visa Sponsorship - San Francisco, United States)

IllTip • San Francisco (CA)

On-site
USD 180,000 - 280,000
AI Evaluation Engineer: RL Environments & Agents
AI Evaluation Engineer: RL Environments & Agents

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 140,000 - 210,000
Enterprise AI Evaluation Infra Engineer — SF On-Site
Enterprise AI Evaluation Infra Engineer — SF On-Site

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 160,000 - 230,000
Agent Evaluation Infrastructure Engineer
Agent Evaluation Infrastructure Engineer

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 160,000 - 230,000
Senior AI Benchmarking & Systems Architect
Senior AI Benchmarking & Systems Architect

Aionia Group • San Francisco (CA)

On-site
USD 130,000 - 220,000
Equity
On-site
AI Evals Engineer – Reward Design & Agent Environments
AI Evals Engineer – Reward Design & Agent Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 120,000 - 180,000
Senior AI Evaluation Backend Engineer - Remote
Senior AI Evaluation Backend Engineer - Remote

Turing • United States

Remote
USD 150,000 - 210,000
AI Benchmarking & Strategy Lead
AI Benchmarking & Strategy Lead

Artificial Analysis • San Francisco (CA)

On-site
USD 180,000 - 260,000
Equity
Competitive compensation