ML Engineer, LLM Eval & Observability (Hybrid)

Gleanwork; San Francisco (CA); Hybrid; USD 200,000 - 300,000; Full time

Benefits offered by this job

Home office improvement stipend

Annual education and wellness stipends

Healthy daily lunches

Job summary

Gleanwork is seeking a Software Engineer to enhance their AI assistant's quality through rigorous evaluation and measurement systems. You'll design datasets and build pipelines while collaborating with engineers across the company.

The ideal candidate has expertise in Go and Python, and at least 2 years of experience. This hybrid role is based in the San Francisco Bay Area, offering competitive compensation and a comprehensive benefits package, including stipends for home office improvements.

Qualifications

• 2+ years of software engineering experience with strong coding skills.
• Strong backend fundamentals in Go and Python; comfortable with distributed data pipelines.
• Experience working with LLM evaluation or reinforcement learning from human feedback.

Responsibilities

• Design and curate evaluation datasets for assistant quality measurement.
• Build and maintain large-scale evaluation pipelines.
• Build LLM-powered judges that score metrics like correctness and completeness.
• Evaluate new models and product changes before they ship.
• Build observability infrastructure for AI agents.

Skills

Software engineering

Go

Python

Machine learning

Natural language processing

Job description

Gleanwork is seeking a Software Engineer to enhance their AI assistant's quality through rigorous evaluation and measurement systems. You'll design datasets and build pipelines while collaborating with engineers across the company.

The ideal candidate has expertise in Go and Python, and at least 2 years of experience. This hybrid role is based in the San Francisco Bay Area, offering competitive compensation and a comprehensive benefits package, including stipends for home office improvements.