Machine Learning Engineer

EducationPals; San Francisco (CA); On-site; USD 150,000 - 189,000; Full time

Benefits offered by this job

Equity

Job summary

EducationPals in San Francisco, CA is seeking a Machine Learning Engineer to contribute to evaluation pipelines for AI systems. The role focuses on defining evaluation scope, building test sets, and deploying LLM-based judging pipelines to assess accuracy, safety, and faithfulness across ML workflows.

The position emphasizes end-to-end evaluation, from dataset design to production monitoring, with a compensation range and equity offered for a competitive package in a dynamic growth environment.

Responsibilities

• Define evaluation scope and use cases for the ML system.
• Build representative test sets capturing real-world diversity and edge cases.
• Write scoring rubrics and metrics for correctness, safety, faithfulness, and task quality.
• Implement LLM-as-Judge pipelines with scalable automation and quality controls.
• Evaluate RAG systems end-to-end, measuring retrieval vs generation quality and faithfulness.
• Perform error analysis and regression testing to catch regressions as systems evolve.
• Extend evaluation to deployment and production, monitoring drift and latency constraints.

Skills

AI fundamentals

Machine learning

Deep learning

Python

Data science

Cloud

Experimentation

LLM evaluation

Job description

Machine Learning Engineer San Francisco, CA Employment Type Compensation $150K – $189K Offers Equity For cash compensation, we set standard ranges for all U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. In order to be compliant with local legislation, as well as to provide greater transparency to candidates, we share salary ranges on all job postings regardless of desired hiring location. Final offer amounts are determined

Relevance is based on skills and concepts, not a guarantee that you qualify.

Covered skills: AI fundamentals Machine learning Deep learning Python Data science Cloud Experimentation LLM evaluation

Skills to close: None listed

Build the skills for this job

Coming-soon courses matched to this posting: outline, waitlist, and skill path.

1. Defining Evaluation Scope and Use Cases

Map product requirements to testable behaviors and identify what good looks like for your specific LLM application.

2. Building Representative Test Sets

Create test datasets that capture real-world diversity, edge cases, and adversarial inputs for your domain.

3. Writing Scoring Rubrics and Metrics

Define explicit, measurable criteria for correctness, safety, faithfulness, and task-specific quality.

4. Implementing LLM-as-Judge Pipelines

Build scalable automated evaluation systems using LLMs as judges, with calibration and quality controls.

5. Evaluating RAG Systems End to End

Separate retrieval quality from generation quality and measure faithfulness, citation, and refusal behavior.

6. Running Error Analysis and Regression Testing

Turn evaluation results into insights and build safety nets that catch regressions when systems change.

7. Evaluation in Deployment and Production

Extend evaluation to live environments, monitor drift, and adapt to device, latency, and user context constraints.

Other verified postings in the same role family or with overlapping skills.