AI Engineer

Momentic; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401(k)

Unlimited PTO

US holidays

Daily lunch

Fully stocked kitchen

Job summary

Momentic is building the definitive verification layer for software: an AI-native testing platform that helps engineering teams catch bugs before they ship. You will join the engineering team in San Francisco to work closely with our CTO on improving the models and agents behind Momentic.

We’re seeking a hands-on AI engineer who can fine-tune models, build evaluations, and turn production data into more reliable, efficient agents.

Qualifications

• 3+ years of software engineering experience with hands-on AI-powered production systems.
• Hands-on experience with supervised fine-tuning, including data preparation and evaluation.
• Experience with prompt/context tuning, evaluation datasets, and LLM observability to diagnose failures and measure improvements.
• Strong engineering fundamentals and ability to write efficient, reliable code.
• Experience using AI agents to build software and manage ambiguity to deliver results.

Responsibilities

• Fine-tune models for long-running agents and diagnose whether failures arise from infrastructure, code, or authorship.
• Build and curate training/evaluation datasets from production workflows.
• Develop LLM evaluations and observability linking offline results to production behavior.
• Tune prompts, context, and model behavior to improve task success, latency, and cost.
• Own technical projects from definition to delivery with cross-team collaboration.

Skills

Software engineering

AI production systems

Fine-tuning

Prompt tuning

LLM observability

Autonomy

Problem solving

Job description

Open - Momentic has a posting for this job. Posted 25 Oct 2026.

This posting

Opens jobs.ashbyhq.com. Talentify has not checked who owns that address. You apply there - Talentify never asks for an account or a CV.

About this job

Momentic is building the definitive verification layer for software: an AI-native testing platform that helps engineering teams catch bugs before they ship. As AI accelerates software development, teams need a reliable way to verify that their products work. Engineering teams at Notion, Webflow, Xero, and Whatnot use Momentic to ship with confidence.

You'll join our engineering team and work closely with our CTO to improve the models and agents behind Momentic. We're looking for a hands-on AI engineer who can fine-tune models, build evaluations, and turn production data into more reliable, efficient agents. Momentic executes more than 1 million tests daily and processes more than 1 billion tokens per hour at peak, giving your improvements a direct impact on the product.

What you'll do

• Fine-tune models for long-running agents, including agents that investigate whether test failures come from infrastructure, recent code changes, or test authorship.
• Build and curate training and evaluation datasets from production workflows, using failure patterns to identify opportunities for improvement.
• Develop LLM evaluations and observability that connect offline results with production behavior and help the team verify improvements.
• Tune prompts, context, and model behavior to improve task success, reliability, latency, token usage, and cost.
• Own technical projects from problem definition through delivery, working with the engineering team to make practical tradeoffs and ship improvements.

You're perfect for this role if…

• You have relevant software engineering experience, typically 3+ years, including hands-on contributions to AI-powered capabilities used in production.
• You have hands-on experience with supervised fine-tuning, including preparing training data and evaluating model behavior.
• You've worked with prompt and context tuning, evaluation datasets, and LLM observability to diagnose failures and measure improvements.
• You have strong engineering fundamentals, code efficiently, and know what you're great at and where you have room to grow.
• You use AI agents to build software, breaking down work, tracking progress, checking results, and redirecting agents when they get stuck.
• You're comfortable with autonomy and ambiguity. You can turn an unclear problem into a plan, manage your own work, and follow through to delivery.

We're interested in the depth of your contributions and what you've shipped. Your experience may come from owning an AI capability within a larger product.

Nice to haves

• Experience with reinforcement learning fine-tuning, RL environments, or LLM-as-judge evaluators.
• Experience with long-running agents, recovery from failed steps, or coordinating multiple agents.
• Experience publishing benchmarks or running rigorous experiments on AI systems.

About us

• Raised a $15M Series A led by Standard Capital, with backing from Y Combinator, FCVC, Transpose Platform, and Dropbox Ventures.
• Our customer base has grown 4× year to date.
• A team of 17 and growing.
• Based in person in San Francisco at 650 5th Street.
• Medical, dental, and vision insurance.
• 401(k).
• Unlimited PTO and sick days, plus company-observed US holidays.
• Daily lunch and a fully stocked kitchen.