Applied Researcher

Enboarder; San Francisco (CA); On-site; USD 150,000 - 600,000; Full time

Benefits offered by this job

Generous salary and equity

Benefits package

US visa sponsorship for exceptional candidates

Job summary

A technology company in San Francisco seeks a Research Engineer to develop their reliability platform for LLM applications. The role focuses on optimization and testing methodologies while emphasizing hands-on implementation and collaboration with clients. Ideal candidates will have a strong publication background and applied engineering experience, with an anticipated salary range from $150,000 to $600,000 annually based on expertise. This position requires a proactive approach towards experimentation and practical applications in machine learning.

Qualifications

• First-author publications in top-tier ML venues.
• Production engineering experience preferred.
• Familiarity with ideas from active learning and synthetic data.

Responsibilities

• Develop optimization and fuzzing methods for LLM systems.
• Implement automated evaluation models and systems.
• Work directly with customers to customize tools.

Skills

First-author publications

Production engineering experience

Familiarity with active learning

Synthetic data

Automated evaluation

Job description

Haize Labs gets LLM apps out of POCs and into production. We eliminate the risk and improve the reliability of LLM apps by haizing them – i.e. rigorously, proactively, and continuously fuzz-testing them.

We are looking for Research Engineers to help develop our reliability platform, with a focus on:

1. Data-efficient alignment of evaluation models
2. Dynamic testing of AI applications
3. Observability and anomaly detection
4. Discrete optimization (with applications in architecture search and automated prompting)

Our work is both intellectually stimulating and practically in high demand. Your work will result in net-new primitives, frameworks, and algorithms for developing robust LLM applications. You work will directly influence how LLM apps are tested, verified, and deployed everywhere. You will directly influence how the world responsibly uses LLMs.

Responsibilities

• Develop optimization, synthetic data generation, and fuzzing methods for breaking LLM systems.
• Implement complex automated evaluation models and systems.
• Go from research idea to code within hours; and iterate quickly on experiments and data.
• Work directly with customers to adapt our tools for different domains.

Qualifications

• First-author publications in top-tier ML venues (NeurIPS, ICML, ICLR, and others).
• Bias for action & experimentation over philosophizing (though sometimes good).
• Not interested in printing papers for papers' sake.
• Some production engineering experience (e.g. ML in an applied setting). No spaghetti research code!
• Some familiarity with ideas from active learning, weak supervision, synthetic data, functional verification, reinforcement learning, reward modeling, automated evaluation. A subset of this is fine.

Annual Salary

$150,000 – $600,000 USD

Logistics

• Location policy: In NYC.
• US visa sponsorship: If you are exceptional, we will sponsor.
• Compensation and Benefits: We provide generous salary, equity, and benefits