AI Behavior Researcher — Impactful AI Evaluation Lead

Transluce; San Francisco (CA); On-site; USD 250,000 - 450,000; Full time

Job summary

Transluce, a fast-moving nonprofit research lab in San Francisco, seeks an AI Behavior Researcher to lead automated evaluations of frontier AI systems, with a focus on wellbeing and user autonomy. You will design pipelines, implement user simulators, and develop robust evaluation rubrics while collaborating with regulators and domain experts.

We value rigorous experimental design and strong Python skills, and we invite applicants across experience levels.

Qualifications

• Experience designing and validating automated AI evaluation methods, such as LLM-as-a-judge systems or multi-turn benchmarks.
• Expertise in quantitative generative AI evaluation and measurement; ability to operationalize social concepts.
• Strong Python proficiency for analysis and tooling.
• Meticulous experimental design with transparency.
• Ability to communicate effectively with researchers and decision makers.

Responsibilities

• Develop novel automated evaluations of AI’s impacts on users, including mental health and decision making.
• Write code to implement and run automated evaluations, such as user simulators or LLM-as-a-judge pipelines.
• Design methods to improve ecological validity of evaluations for specific populations.
• Write and revise rubrics to evaluate model behaviors related to wellbeing and decision making.
• Collaborate with scientists and research engineers to productionize best practices in AI behavioral evaluation.

Skills

Quantitative AI evaluation

Python proficiency

Experimental design

Communication skills

Tools

LLM-as-a-judge pipelines

Job description

Transluce, a fast-moving nonprofit research lab in San Francisco, seeks an AI Behavior Researcher to lead automated evaluations of frontier AI systems, with a focus on wellbeing and user autonomy. You will design pipelines, implement user simulators, and develop robust evaluation rubrics while collaborating with regulators and domain experts.

We value rigorous experimental design and strong Python skills, and we invite applicants across experience levels.