Founding Engineer: AI Environments & Evaluation

Wwshemi; San Francisco (CA); On-site; USD 200,000 - 300,000; Full time

Benefits offered by this job

Founding equity

Visa sponsorship

Job summary

Wwshemi is seeking a founding engineer to design synthetic company environments and develop evaluation frameworks for AI agents. The role blends research with production readiness to prove agent capabilities in real-world company tasks.

You will build benchmarks, graders, and simulation environments, ensuring high fidelity tools and scalable evaluation methods. Visa sponsorship is available, and equity is part of the package.

Qualifications

• Hands-on research experience in synthetic data pipelines, benchmarks, evaluations, or model post-training.
• Experience building environments or benchmarks used to train or evaluate agents at AI labs.
• Experience implementing graders or evaluation systems for AI agents.

Responsibilities

• Design synthetic company environments with realistic data for agents to use over time.
• Build graders and evaluation systems for advanced agents and cases with limited expert oversight.
• Develop benchmarks that correlate with real-world production performance.
• Identify and mitigate evaluation failure modes such as reward hacking and benchmark gaming.
• Simulate software tools (inboxes, CRMs) to support large-scale agent interactions.

Skills

Synthetic data pipelines

Benchmarks design

Evaluation systems

AI agent evaluation

Research experience

Education

Degree from selective university

Founder program participation

Tools

CRMs

Email inbox simulation

Job description

Wwshemi is seeking a founding engineer to design synthetic company environments and develop evaluation frameworks for AI agents. The role blends research with production readiness to prove agent capabilities in real-world company tasks.

You will build benchmarks, graders, and simulation environments, ensuring high fidelity tools and scalable evaluation methods. Visa sponsorship is available, and equity is part of the package.