AI Engineer (REMOTE)

EducationPals; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

EducationPals is seeking an AI Engineer who builds, trains, and evaluates machine learning models with hands-on ownership from data to reporting. The role blends knowledge engineering with applied reinforcement learning and requires collaboration with domain experts while communicating results to stakeholders.

The position emphasizes practical RL experience, strong engineering discipline, and clear results presentation.

Qualifications

• 1. Defining Evaluation Scope and Use Cases: Map product requirements to testable behaviors and identify good outcomes for your LLM application.
• 2. Building Representative Test Sets: Create datasets that reflect real-world diversity, edge cases, and adversarial inputs.
• 3. Writing Scoring Rubrics and Metrics: Define explicit criteria for correctness, safety, faithfulness, and task-specific quality.
• 4. Implementing LLM-as-Judge Pipelines: Build scalable automated evaluation systems with calibration and quality controls.
• 5. Evaluating RAG Systems End to End: Separate retrieval quality from generation quality and measure faithfulness and citations.
• 6. Running Error Analysis and Regression Testing: Turn results into insights and catch regressions when models evolve.
• 7. Evaluation in Deployment and Production: Extend evaluation to live contexts, monitor drift, latency, and user context constraints.

Responsibilities

• Map product requirements to testable behaviors and identify good outcomes for your LLM application.
• Create test datasets that capture real-world diversity, edge cases, and adversarial inputs for your domain.
• Define explicit, measurable criteria for correctness, safety, faithfulness, and task-specific quality.
• Build scalable automated evaluation systems using LLMs as judges, with calibration and quality controls.
• Separate retrieval quality from generation quality and measure faithfulness, citation, and refusal behavior.
• Turn evaluation results into insights and build safety nets that catch regressions when systems change.
• Extend evaluation to live environments, monitor drift, and adapt to device, latency, and user context constraints.

Job description

The AI Engineer is a hands-on individual contributor who builds, trains, and evaluates machine learning models. The role combines knowledge engineering and data analysis with applied reinforcement learning, and owns work end to end, from data and experiment design through training, evaluation, and reporting. The ideal candidate brings practical reinforcement learning experience, strong engineering discipline, and the ability to collaborate closely with domain experts and communicate results to t

• Special Considerations and Prerequisites
• EnableComp Home Page

Courses relevant to this posting

Relevance is based on skills and concepts, not a guarantee that you qualify.

Covered skills: AI fundamentals Machine learning LLM evaluation

Skills to close: None listed

Build the skills for this job

Coming-soon courses matched to this posting: outline, waitlist, and skill path.

1. Defining Evaluation Scope and Use Cases

Map product requirements to testable behaviors and identify what good looks like for your specific LLM application.

2. Building Representative Test Sets

Create test datasets that capture real-world diversity, edge cases, and adversarial inputs for your domain.

3. Writing Scoring Rubrics and Metrics

Define explicit, measurable criteria for correctness, safety, faithfulness, and task‑specific quality.

4. Implementing LLM-as-Judge Pipelines

Build scalable automated evaluation systems using LLMs as judges, with calibration and quality controls.

5. Evaluating RAG Systems End to End

Separate retrieval quality from generation quality and measure faithfulness, citation, and refusal behavior.

6. Running Error Analysis and Regression Testing

Turn evaluation results into insights and build safety nets that catch regressions when systems change.

7. Evaluation in Deployment and Production

Extend evaluation to live environments, monitor drift, and adapt to device, latency, and user context constraints.

Want the full course when it launches? Join the waitlist and we will notify you.

Want the full course when it launches? Join the waitlist and we will notify you.

Email address

By joining, you agree to receive product updates and marketing emails about this course. Privacy Policy

Other verified postings in the same role family or with overlapping skills.