ML Engineer: LLM Evaluation & Observability

Gleanwork; Mountain View (CA); Hybrid; USD 200,000 - 300,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Home office improvement stipend

Annual education stipend

Wellness stipend

Daily healthy lunches

Job summary

Gleanwork is looking for a software engineer focused on AI assistant evaluation in Mountain View, CA. You will design evaluation datasets and build pipelines to measure assistant quality, collaborating closely with cross-functional teams.

The ideal candidate has strong coding skills in Go and Python, and experience with LLM evaluation. The position offers a competitive salary between $200,000 and $300,000, with hybrid work options and comprehensive benefits including health coverage and stipends for home office improvements and education.

Qualifications

• 2+ years of software engineering experience with strong coding skills.
• Strong backend fundamentals in Go and Python.
• Comfortable with distributed data pipelines.
• Experience with LLM evaluation or reinforcement learning.
• Analytically rigorous in predicting user experience.
• Strong team player in a cross-functional environment.

Responsibilities

• Design and curate evaluation datasets for assistant behavior.
• Build large-scale evaluation pipelines for quality measurement.
• Develop LLM-powered judges for scoring assistant metrics.
• Evaluate new models and product changes before launches.
• Build observability infrastructure for AI agents.
• Collaborate with engineers to enhance evaluation processes.

Skills

Software engineering experience

Backend fundamentals in Go and Python

Experience with LLM evaluation

Analytical rigor

Customer-focused mindset

Job description

Gleanwork is looking for a software engineer focused on AI assistant evaluation in Mountain View, CA. You will design evaluation datasets and build pipelines to measure assistant quality, collaborating closely with cross-functional teams.

The ideal candidate has strong coding skills in Go and Python, and experience with LLM evaluation. The position offers a competitive salary between $200,000 and $300,000, with hybrid work options and comprehensive benefits including health coverage and stipends for home office improvements and education.