LLM Evaluation Engineer — ML, RAG & Production

EducationPals; San Francisco (CA); On-site; USD 150,000 - 189,000; Full time

Benefits offered by this job

Equity

Job summary

EducationPals in San Francisco, CA is seeking a Machine Learning Engineer to contribute to evaluation pipelines for AI systems. The role focuses on defining evaluation scope, building test sets, and deploying LLM-based judging pipelines to assess accuracy, safety, and faithfulness across ML workflows.

The position emphasizes end-to-end evaluation, from dataset design to production monitoring, with a compensation range and equity offered for a competitive package in a dynamic growth environment.

Responsibilities

• Define evaluation scope and use cases for the ML system.
• Build representative test sets capturing real-world diversity and edge cases.
• Write scoring rubrics and metrics for correctness, safety, faithfulness, and task quality.
• Implement LLM-as-Judge pipelines with scalable automation and quality controls.
• Evaluate RAG systems end-to-end, measuring retrieval vs generation quality and faithfulness.
• Perform error analysis and regression testing to catch regressions as systems evolve.
• Extend evaluation to deployment and production, monitoring drift and latency constraints.

Skills

AI fundamentals

Machine learning

Deep learning

Python

Data science

Cloud

Experimentation

LLM evaluation

Job description

EducationPals in San Francisco, CA is seeking a Machine Learning Engineer to contribute to evaluation pipelines for AI systems. The role focuses on defining evaluation scope, building test sets, and deploying LLM-based judging pipelines to assess accuracy, safety, and faithfulness across ML workflows.

The position emphasizes end-to-end evaluation, from dataset design to production monitoring, with a compensation range and equity offered for a competitive package in a dynamic growth environment.