AI Engineer | RL & End-to-End Model Evaluation

EducationPals; United States; A distancia; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

EducationPals is seeking an AI Engineer who builds, trains, and evaluates machine learning models with hands-on ownership from data to reporting. The role blends knowledge engineering with applied reinforcement learning and requires collaboration with domain experts while communicating results to stakeholders.

The position emphasizes practical RL experience, strong engineering discipline, and clear results presentation.

Formación

• 1. Defining Evaluation Scope and Use Cases: Map product requirements to testable behaviors and identify good outcomes for your LLM application.
• 2. Building Representative Test Sets: Create datasets that reflect real-world diversity, edge cases, and adversarial inputs.
• 3. Writing Scoring Rubrics and Metrics: Define explicit criteria for correctness, safety, faithfulness, and task-specific quality.
• 4. Implementing LLM-as-Judge Pipelines: Build scalable automated evaluation systems with calibration and quality controls.
• 5. Evaluating RAG Systems End to End: Separate retrieval quality from generation quality and measure faithfulness and citations.
• 6. Running Error Analysis and Regression Testing: Turn results into insights and catch regressions when models evolve.
• 7. Evaluation in Deployment and Production: Extend evaluation to live contexts, monitor drift, latency, and user context constraints.

Responsabilidades

• Map product requirements to testable behaviors and identify good outcomes for your LLM application.
• Create test datasets that capture real-world diversity, edge cases, and adversarial inputs for your domain.
• Define explicit, measurable criteria for correctness, safety, faithfulness, and task-specific quality.
• Build scalable automated evaluation systems using LLMs as judges, with calibration and quality controls.
• Separate retrieval quality from generation quality and measure faithfulness, citation, and refusal behavior.
• Turn evaluation results into insights and build safety nets that catch regressions when systems change.
• Extend evaluation to live environments, monitor drift, and adapt to device, latency, and user context constraints.

Descripción del empleo

EducationPals is seeking an AI Engineer who builds, trains, and evaluates machine learning models with hands-on ownership from data to reporting. The role blends knowledge engineering with applied reinforcement learning and requires collaboration with domain experts while communicating results to stakeholders.

The position emphasizes practical RL experience, strong engineering discipline, and clear results presentation.