Applied AI Software Engineer: LLM Agent Evaluation Lead

Canvas Construction; San Francisco (CA); Hybrid; USD 300,000 - 400,000; Full time

Benefits offered by this job

Competitive Salary & Equity Package

Health Insurance

Home Office Stipend

401k

Paid Maternity/Paternity Leave

Flexible/unlimited PTO

Job summary

A healthcare technology company based in California is hiring an Applied AI Software Engineer. The role involves leading evaluations of LLM-based systems and collaborating with multidisciplinary teams to ensure robust AI agents are utilized in healthcare. Ideal candidates will have extensive experience with machine learning, foundation model APIs, and Python engineering. The position offers a competitive salary range between $300,000 and $400,000, along with a comprehensive benefits package including health insurance and flexible PTO.

Qualifications

• 5+ years of experience in applied machine learning or AI engineering, focusing on evaluation.
• Proficiency with foundation model APIs and experience orchestrating complex agent behaviors.
• Experience designing and running high-throughput evaluation pipelines.

Responsibilities

• Design and execute evaluation plans for LLM-based agents.
• Build test harnesses for model behavior validation.
• Run experiments across multiple models to determine optimal configurations.

Skills

Evaluating LLM-based systems

Foundation model APIs

Experiment tracking

Python engineering

Education

5+ years in applied machine learning or AI engineering

Tools

SQL

Data management tools

Job description

A healthcare technology company based in California is hiring an Applied AI Software Engineer. The role involves leading evaluations of LLM-based systems and collaborating with multidisciplinary teams to ensure robust AI agents are utilized in healthcare. Ideal candidates will have extensive experience with machine learning, foundation model APIs, and Python engineering. The position offers a competitive salary range between $300,000 and $400,000, along with a comprehensive benefits package including health insurance and flexible PTO.