Founding Engineer, Environments and Data (Visa Sponsorship - San Francisco, United States)

IllTip

San Francisco (CA)

On-site

USD 180,000 - 280,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

IllTip is seeking a founding engineer in San Francisco to design and build evaluation environments and benchmarks for AI agents, ensuring agent performance reaches real-world production readiness. This is a founding role with a research-forward, hands-on engineering focus in a fast-growing AI startup.

You will create synthetic company environments, build graders, and develop benchmarks that align with practical production outcomes, while addressing challenges like reward hacking and benchmark

Qualifications

  • Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
  • Experience building environments or benchmarks used to train or evaluate agents, and designing graders or evaluation systems for AI agents.
  • Experience simulating complex software systems for agent tool use, and finding benchmark flaws or fixing reward hacking.

Responsibilities

  • Design synthetic company environments with realistic email, records, and files that support extended agent use.
  • Build robust graders and evaluation systems that can judge agent output even where human reviewers are not expert enough to do so.
  • Develop benchmarks that correlate with real-world production performance.
  • Identify and fix evaluation failures, including reward hacking and benchmark gaming.
  • Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.

Skills

Synthetic data pipelines
Benchmarks & evaluations
Agent tool environments
Research & engineering

Job description

Visa sponsorship available. A fast-growing AI-focused company is hiring a founding engineer to build evaluation environments and benchmarks that measure whether AI agents can complete real company work at or above human level. This is a founding role in a small team, with a strong emphasis on research and engineering rather than infrastructure alone. The work shapes how agent competence is measured and whether agents are ready for production use.


Responsibilities


  • Design synthetic company environments with realistic email, records, and files that support extended agent use.

  • Build robust graders and evaluation systems that can judge agent output even where human reviewers are not expert enough to do so.

  • Develop benchmarks that correlate with real-world production performance.

  • Identify and fix evaluation failures, including reward hacking and benchmark gaming.

  • Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.


Requirements


  • Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training.

  • Experience building environments or benchmarks used to train or evaluate agents, and designing graders or evaluation systems for AI agents.

  • Experience simulating complex software systems for agent tool use, and finding benchmark flaws or fixing reward hacking.

  • At least 1 to 2 years at a fast-growing or AI-focused startup, with relevant AI research or engineering experience.

  • Strong startup experience (such as founding a company or joining an accelerator) or high-level competition experience in areas such as mathematics, programming, debate, or games.

  • A clear belief that AI agents can outperform humans at many computer-based tasks.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding Engineer, AI Agents (Visa Sponsorship - San Francisco, USA)
Founding Engineer, AI Agents (Visa Sponsorship - San Francisco, USA)

IllTip • San Francisco (CA)

On-site
USD 150,000 - 230,000
Founding Engineer – RL Environments & Data
Founding Engineer – RL Environments & Data

CGS • San Francisco (CA)

On-site
USD 150,000 - 230,000
Unlimited AI spend
Daily lunch provided
Equity—meaningful
Founding AI Evaluation Engineer
Founding AI Evaluation Engineer

IllTip • San Francisco (CA)

On-site
USD 180,000 - 280,000
Lead Research Engineer, Data Quality (Visa Sponsorship - United States)
Lead Research Engineer, Data Quality (Visa Sponsorship - United States)

IllTip • United States

On-site
USD 180,000 - 280,000
Research Engineer – Training and Environment Infrastructure
Research Engineer – Training and Environment Infrastructure

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 120,000 - 180,000
Health, dental and vision coverage
Hybrid work arrangement
Founding Engineer
Founding Engineer

Success Matcher Recruitment • New York (NY)

On-site
USD 120,000 - 210,000
Founding Research Engineer – RL Environments & Evals
Founding Research Engineer – RL Environments & Evals

Rnb Consultancy • San Francisco (CA)

On-site
USD 150,000 - 200,000
Base salary $150k–$200k
Equity 0.25–0.5%
Profit sharing up to 4% of project
+1
Agent Evaluation Infrastructure Engineer
Agent Evaluation Infrastructure Engineer

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 160,000 - 230,000
Founding Engineer – Full-Stack
Founding Engineer – Full-Stack

CGS • San Francisco (CA)

On-site
USD 180,000 - 240,000
Founding equity
Unlimited AI spend
Daily lunch
Member of Technical Staff, Product
Member of Technical Staff, Product

Raydar • San Francisco (CA)

On-site
USD 200,000 - 400,000
Equity
Visa sponsorships