Staff Machine Learning Engineer

EducationPals; United States; Remote; USD 230,000 - 300,000; Full time

Benefits offered by this job

Bonus

Equity

Job summary

EducationPals is hiring a Staff Machine Learning Engineer to set the technical direction for its agent systems. This remote role focuses on end-to-end model and systems work that ships to large enterprise customers, not a thin layer over someone else’s API.

You will design multi-step LLM systems, oversee evaluation, and collaborate across teams to deliver robust AI capabilities with a strong emphasis on safety, reliability, and scalable performance.

Qualifications

• Map product requirements to testable behaviours and identify what good looks like for your specific LLM application.
• Define evaluation strategies for model safety, faithfulness, and reliability.
• Develop end-to-end evaluation plans for multi-step LLM systems.
• Establish metrics and benchmarks to compare retrieval, planning, and tool use.
• Create scalable evaluation pipelines using LLMs as judges with calibration.
• Measure performance across edge cases, drift, and production deployment.
• Coordinate cross-functional efforts to implement robust evaluation in production.

Responsibilities

• Define evaluation scope and use cases for your LLM applications.
• Build representative test datasets that capture real-world diversity and edge cases.
• Write scoring rubrics and metrics for correctness, safety, faithfulness, and quality.
• Implement LLM-as-Judge pipelines for scalable automated evaluation with calibration.
• Evaluate RAG systems end-to-end, separating retrieval and generation quality.
• Run error analysis and regression testing to catch regressions in systems.
• Extend evaluation to deployment, monitor drift, and adapt to context and latency.

Skills

AI fundamentals

Machine learning

Deep learning

GenAI

Cloud

LLM evaluation

Job description

Staff Machine Learning Engineer Employment Type Location Type OverviewApplication Location: Remote (US) Pay: $230,000–$300,000 + bonus + equity A late-stage, venture-backed enterprise AI company is hiring a Staff ML Engineer to set the technical direction for its agent systems. This is model and systems work that ships to large enterprise customers, not a thin layer over someone else's API. What you'll do Design multi-step LLM systems: reasoning, retrieval, planning and tool use Own evaluation,

Relevance is based on skills and concepts, not a guarantee that you qualify.

Covered skills: AI fundamentals Machine learning Deep learning GenAI Cloud LLM evaluation

Skills to close: None listed

Build the skills for this job

Coming-soon courses matched to this posting: outline, waitlist, and skill path.

1. Defining Evaluation Scope and Use Cases

Map product requirements to testable behaviours and identify what good looks like for your specific LLM application.

2. Building Representative Test Sets

Create test datasets that capture real-world diversity, edge cases, and adversarial inputs for your domain.

3. Writing Scoring Rubrics and Metrics

Define explicit, measurable criteria for correctness, safety, faithfulness, and task‑specific quality.

4. Implementing LLM-as-Judge Pipelines

Build scalable automated evaluation systems using LLMs as judges, with calibration and quality controls.

5. Evaluating RAG Systems End to End

Separate retrieval quality from generation quality and measure faithfulness, citation, and refusal behavior.

6. Running Error Analysis and Regression Testing

Turn evaluation results into insights and build safety nets that catch regressions when systems change.

7. Evaluation in Deployment and Production

Extend evaluation to live environments, monitor drift, and adapt to device, latency, and user context constraints.

This job posting may span more than one career level. In addition to base salary, this job is eligible to receive equity in the form of res…