Agent Operations Engineer (LLMOps)

Momcozy

California

On-site

USD 120,000 - 160,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Momcozy is building local engineering teams in the United States and is seeking an Agent Operations Engineer to own production stability and ongoing operations of our AI agent products in the U.S. You will manage prompt versions, knowledge base updates, and tool calling configuration while collaborating with product, algorithm, and engineering teams.

This hands‑on role requires deep understanding of LLM applications, incident response, staged rollouts, and safety/compliance checks.

Qualifications

  • 3+ years in software development, operations, or technical support, with AI / LLM project experience.
  • Solid understanding of LLM application engineering: prompt engineering, RAG, function calling, streaming, and context management.
  • Production operations experience with monitoring, alerting, log analysis, troubleshooting, and incident response.
  • Proficient in Python for writing operations scripts, data processing, and automation.
  • Working knowledge of LLM fundamentals and common issues (hallucination, latency variance, cost control, safety boundaries).
  • Experience with A/B testing and evaluation, including experiment design and result analysis.
  • Strong analytical and problem‑solving skills in complex systems.
  • Professional fluency in both English and Chinese, including technical discussions, meetings, and written documentation.
  • Ability to work effectively across cultures, countries, and time zones.

Responsibilities

  • Own production operations of AI agents, including model service configuration, prompt versioning, knowledge base updates, and tool calling configuration
  • Build an agent performance monitoring system tracking response quality, TTFT, total latency, token cost, and user satisfaction
  • Lead rapid incident triage and root‑cause analysis, coordinating with algorithm and engineering teams on fixes and systemic improvements
  • Manage staged rollouts, A/B testing, and evaluation workflows for agent iterations
  • Maintain and optimize prompt templates, tool configurations, and knowledge base content to continuously improve answer quality
  • Establish a Bad Case collection, classification, analysis, and closed‑loop tracking mechanism with regular review reports
  • Own safety and compliance checks for agents, including content safety, privacy protection, access control, and privilege escalation prevention
  • Write operations documentation, runbooks, and emergency response procedures
  • Collaborate with product teams to translate user feedback and production data into agent improvement requirements
  • Adopt AI Coding tools to automate operations workflows and improve efficiency

Skills

Python
LLM experience
Operations
Monitoring
Incident response
AB testing
Cross-cultural communication
English & Chinese fluency

Job description

We are building local engineering teams in Singapore and the United States and are looking for an Agent Operations Engineer with strong AI application knowledge and solid operations engineering skills. You will own the production stability, quality, and ongoing operations of our AI agent products. Your responsibilities include prompt management, knowledge base maintenance, model configuration, performance monitoring, incident response, and staged rollouts—working closely with product, algorithm, and engineering teams to ensure our AI agents run reliably and effectively in production.

This is a hands‑on role that blends technical depth with product awareness. We expect you to deeply understand the engineering characteristics of LLM applications, establish standardized agent operations practices, and maintain strong sensitivity to user experience and business outcomes.

Responsibilities
  • Own production operations of AI agents, including model service configuration, prompt versioning, knowledge base updates, and tool calling configuration
  • Build an agent performance monitoring system tracking response quality, TTFT, total latency, token cost, and user satisfaction
  • Lead rapid incident triage and root‑cause analysis, coordinating with algorithm and engineering teams on fixes and systemic improvements
  • Manage staged rollouts, A/B testing, and evaluation workflows for agent iterations
  • Maintain and optimize prompt templates, tool configurations, and knowledge base content to continuously improve answer quality
  • Establish a Bad Case collection, classification, analysis, and closed‑loop tracking mechanism with regular review reports
  • Own safety and compliance checks for agents, including content safety, privacy protection, access control, and privilege escalation prevention
  • Write operations documentation, runbooks, and emergency response procedures
  • Collaborate with product teams to translate user feedback and production data into agent improvement requirements
  • Adopt AI Coding tools to automate operations workflows and improve efficiency
Qualifications
  • 3+ years in software development, operations, or technical support, with AI / LLM project experience
  • Solid understanding of LLM application engineering: prompt engineering, RAG, function calling, streaming, and context management
  • Production operations experience with monitoring, alerting, log analysis, troubleshooting, and incident response
  • Proficient in Python for writing operations scripts, data processing, and automation
  • Working knowledge of LLM fundamentals and common issues (hallucination, latency variance, cost control, safety boundaries)
  • Experience with A/B testing and evaluation, including experiment design and result analysis
  • Strong analytical and problem‑solving skills in complex systems
  • Professional fluency in both English and Chinese, including technical discussions, meetings, and written documentation
  • Ability to work effectively across cultures, countries, and time zones
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Agent Operations Engineer
Agent Operations Engineer

Momcozy • Los Angeles (CA)

On-site
USD 110,000 - 170,000
Algorithm Engineer (AI Agent)
Algorithm Engineer (AI Agent)

Momcozy • California

On-site
USD 150,000 - 190,000
AI Engineer, Agent
AI Engineer, Agent

Eloquent AI • San Francisco (CA)

On-site
USD 140,000 - 210,000
Senior AI Agent Engineer
Senior AI Agent Engineer

EPAM Systems • United States

On-site
USD 150,000 - 210,000
Lead AI Agent Engineer
Lead AI Agent Engineer

EPAM Systems • United States

On-site
USD 180,000 - 240,000
Software Development Engineer 3- #26-27306
Software Development Engineer 3- #26-27306

Us Tech Solutions • Austin (TX)

On-site
USD 140,000 - 210,000
Agentic AI Engineer
Agentic AI Engineer

Tactical Edge • Washington

On-site
USD 150,000 - 210,000
AI Engineer
AI Engineer

Altitude Technology Solutions Inc • Dallas (TX)

Hybrid
USD 140,000 - 190,000
AI Agents Applied Research/Engineering Lead - Vice President
AI Agents Applied Research/Engineering Lead - Vice President

JPMorgan Chase & Co. • New York (NY)

On-site
USD 180,000 - 240,000
AI Agent Engineer
AI Agent Engineer

Ethereum Technologies LLC • California

On-site
USD 140,000 - 210,000