Staff ML Engineer - Remote, Equity, Enterprise AI Systems

EducationPals; United States; Remote; USD 230,000 - 300,000; Full time

Benefits offered by this job

Bonus

Equity

Job summary

EducationPals is hiring a Staff Machine Learning Engineer to set the technical direction for its agent systems. This remote role focuses on end-to-end model and systems work that ships to large enterprise customers, not a thin layer over someone else’s API.

You will design multi-step LLM systems, oversee evaluation, and collaborate across teams to deliver robust AI capabilities with a strong emphasis on safety, reliability, and scalable performance.

Qualifications

• Map product requirements to testable behaviours and identify what good looks like for your specific LLM application.
• Define evaluation strategies for model safety, faithfulness, and reliability.
• Develop end-to-end evaluation plans for multi-step LLM systems.
• Establish metrics and benchmarks to compare retrieval, planning, and tool use.
• Create scalable evaluation pipelines using LLMs as judges with calibration.
• Measure performance across edge cases, drift, and production deployment.
• Coordinate cross-functional efforts to implement robust evaluation in production.

Responsibilities

• Define evaluation scope and use cases for your LLM applications.
• Build representative test datasets that capture real-world diversity and edge cases.
• Write scoring rubrics and metrics for correctness, safety, faithfulness, and quality.
• Implement LLM-as-Judge pipelines for scalable automated evaluation with calibration.
• Evaluate RAG systems end-to-end, separating retrieval and generation quality.
• Run error analysis and regression testing to catch regressions in systems.
• Extend evaluation to deployment, monitor drift, and adapt to context and latency.

Skills

AI fundamentals

Machine learning

Deep learning

GenAI

Cloud

LLM evaluation

Job description

EducationPals is hiring a Staff Machine Learning Engineer to set the technical direction for its agent systems. This remote role focuses on end-to-end model and systems work that ships to large enterprise customers, not a thin layer over someone else’s API.

You will design multi-step LLM systems, oversee evaluation, and collaborate across teams to deliver robust AI capabilities with a strong emphasis on safety, reliability, and scalable performance.