Senior Machine Learning Engineer, Agentic Evals and Self-Improvement (Nova)

Iterable, Inc.; New Boston (IL); On-site; USD 150,000 - 250,000; Full time

Job summary

Iterable is seeking a Senior Machine Learning Engineer for the Nova team to lead evaluation and automatic optimization of models and agents. You’ll measure performance, diagnose failures, and drive improvements across prompts, context, tools, and harnesses in production.

You will own end‑to‑end experimentation, data pipelines, and integration with Nova retrieval and product teams, shaping AI direction at Iterable. We value growth mindset and ongoing learning across roles.

Qualifications

• 5+ years of hands‑on MLE experience deploying complex ML systems at scale.
• Master's or PhD in Computer Science, Machine Learning, AI, or related field.
• Strong Python skills with readable, modular code and experience with modern ML frameworks.
• Experience evaluating LLMs/agents, including multi‑turn behavior and tool use.
• Experience building automated evaluation: LLM judges, reward models, or benchmark infra.
• Experience with prompt/context optimization, post‑training, or reinforcement learning.

Responsibilities

• Build evaluation systems for models and agents: benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
• Create a loop that converts failures from diagnosis to fixes and measurable improvements.
• Turn evaluation results into automatic optimization of prompts, context, tools, rubrics, and agent harnesses.
• Build synthetic data pipelines for evaluation and post‑training with emphasis on quality, privacy, and reproducibility.
• Apply recent research (LLM-as-judge, reward modeling, test‑time search) to production.
• Mentor engineers on the team and help define the technical direction for Nova.

Skills

MLE in production

Python

ML frameworks

LLM evaluation

Prompt optimization

Reinforcement learning

Cross‑team collaboration

Strong communication

Education

Master's or PhD in CS/ML/AI

Job description

Iterable is the AI customer engagement platform, built for enterprise scale, loved by teams, and trusted by global brands like Redfin, SeatGeek, Priceline, Calm, and Box. Our platform empowers organizations to activate customer data from any source, design seamless cross-channel experiences, and optimize engagement with Nova Intelligence, the AI layer of our platform, all with enterprise‑grade security and compliance. Today, nearly 1,200 brands across 50+ countries rely on Iterable to drive growth, deepen customer relationships, and deliver joyful customer experiences. Let’s shape the future of customer engagement together!

Location: US

About The Role

As a high‑tech marketing platform, Iterable works with thousands of large‑scale and diverse enterprise customers. Iterable Nova is our intelligence layer. It helps customer experience teams adapt in real time and improve every interaction. As a Senior Machine Learning Engineer on Nova, you’ll lead evaluation and automatic optimization for Nova’s models and agents. That means measuring how they perform, finding where they fail, and using those findings to automatically improve prompts, context, tools, and agent harnesses. You will work with the Nova retrieval and product teams to connect real interactions to measurable improvements.

You will have real autonomy: from research and experimentation through to systems running reliably in production, you’ll own this work and help shape the direction of AI at Iterable.

One of our core values is a growth mindset, and Iterable is a company where everyone can grow. If this role excites you, please apply— we value applicants for the skills they bring beyond a job description.

What You'll Do

• Build evaluation systems for models and agents: benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
• Create a loop that takes failures from diagnosis to fix to measured improvement.
• Turn evaluation results into automatic optimization of prompts, context, tools, rubrics, and agent harnesses.
• Build synthetic data pipelines for evaluation and post‑training, with attention to quality, privacy, and reproducibility.
• Apply recent research (LLM‑as‑judge, reward modeling, test‑time search, post‑training) to production.
• Mentor engineers on the team and help set the technical direction for Nova.

What We're Looking For

• 5+ years of hands‑on MLE experience putting complex ML systems into production at scale, such as production ML pipelines or large‑scale experimentation platforms.
• Master's or PhD in Computer Science, Machine Learning, AI, or a related field.
• You keep up with the latest research and focus on applying it: shipping reliable changes that measurably improve the product experience for customers.
• Strong Python skills with an emphasis on readable, modular, and maintainable code, plus experience with modern ML frameworks.
• Deep experience evaluating LLMs or agents, including multi‑turn behavior, tool use, and reasoning.
• Experience building automated evaluation, such as LLM judges, reward models, or benchmark infrastructure.
• Experience with at least one of: prompt or context optimization, post‑training, preference optimization, or reinforcement learning.
• Strong communication and cross‑team collaboration.

Bonus points

• Experience with LLM APIs from multiple providers (Anthropic, OpenAI, Google), including tool calling and prompt caching.
• Experience with automatic prompt optimization and error analysis.

Iterable is an Equal Employment Opportunity employer that proudly pursues and hires a diverse workforce. Iterable does not make hiring or employment decisions on the basis of race, color, religion or religious belief, ethnic or national origin, nationality, sex, gender, gender‑identity, sexual orientation, disability, age, military or veteran status, or any other basis protected by applicable local, state, or federal laws or prohibited by Company policy. Iterable also strives for a healthy and safe workplace and strictly prohibits harassment of any kind. Pursuant to the San Francisco Fair Chance Ordinance and other similar state laws and local ordinances, and its internal policy, Iterable will also consider for employment qualified applicants with arrest and conviction records.

Compensation Range: $150K - $250K