Senior ML Engineer, Nova AI Platform: Real‑World Impact

Iterable, Inc.; New Boston (IL); On-site; USD 150,000 - 250,000; Full time

Job summary

Iterable is seeking a Senior Machine Learning Engineer for the Nova team to lead evaluation and automatic optimization of models and agents. You’ll measure performance, diagnose failures, and drive improvements across prompts, context, tools, and harnesses in production.

You will own end‑to‑end experimentation, data pipelines, and integration with Nova retrieval and product teams, shaping AI direction at Iterable. We value growth mindset and ongoing learning across roles.

Qualifications

• 5+ years of hands‑on MLE experience deploying complex ML systems at scale.
• Master's or PhD in Computer Science, Machine Learning, AI, or related field.
• Strong Python skills with readable, modular code and experience with modern ML frameworks.
• Experience evaluating LLMs/agents, including multi‑turn behavior and tool use.
• Experience building automated evaluation: LLM judges, reward models, or benchmark infra.
• Experience with prompt/context optimization, post‑training, or reinforcement learning.

Responsibilities

• Build evaluation systems for models and agents: benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
• Create a loop that converts failures from diagnosis to fixes and measurable improvements.
• Turn evaluation results into automatic optimization of prompts, context, tools, rubrics, and agent harnesses.
• Build synthetic data pipelines for evaluation and post‑training with emphasis on quality, privacy, and reproducibility.
• Apply recent research (LLM-as-judge, reward modeling, test‑time search) to production.
• Mentor engineers on the team and help define the technical direction for Nova.

Skills

MLE in production

Python

ML frameworks

LLM evaluation

Prompt optimization

Reinforcement learning

Cross‑team collaboration

Strong communication

Education

Master's or PhD in CS/ML/AI

Job description

Iterable is seeking a Senior Machine Learning Engineer for the Nova team to lead evaluation and automatic optimization of models and agents. You’ll measure performance, diagnose failures, and drive improvements across prompts, context, tools, and harnesses in production.

You will own end‑to‑end experimentation, data pipelines, and integration with Nova retrieval and product teams, shaping AI direction at Iterable. We value growth mindset and ongoing learning across roles.