Staff Software Engineer, Safeguards Evaluations & Trust

Menlo Ventures; San Francisco (CA); Hybrid; USD 320,000 - 485,000; Full time

Benefits offered by this job

Flexible working hours

Generous vacation and parental leave

Office space for collaboration

Job summary

Menlo Ventures in San Francisco seeks an expert to build evaluation infrastructure for safety systems. This role involves designing experiments to measure an investigative agent's performance, building real datasets, and enhancing automated abuse detection systems.

The ideal candidate has a solid background in Python, LLMs, and data analysis, and they will produce high-quality datasets representing real-world misuse scenarios.

Qualifications

• Proficiency in Python and comfort working across the stack.
• Ability to translate ambiguous problems into concrete experiments.
• Experience working with LLMs and their failure modes.

Responsibilities

• Build and own evaluation harnesses for agentic investigation systems.
• Construct high-quality eval datasets representing real-world misuse.
• Analyze gaps in measurement and evolve evaluations over time.

Skills

Python proficiency

Data analysis

Experience with LLMs

Data pipeline management

Education

Bachelor’s degree or equivalent

Tools

Data augmentation techniques

Distributed systems

Job description

Menlo Ventures in San Francisco seeks an expert to build evaluation infrastructure for safety systems. This role involves designing experiments to measure an investigative agent's performance, building real datasets, and enhancing automated abuse detection systems.

The ideal candidate has a solid background in Python, LLMs, and data analysis, and they will produce high-quality datasets representing real-world misuse scenarios.