AI Engineer, Agents & Quality - Build Tools & Evaluation

Gold Rose AI; United States; Hybrid; USD 120,000 - 180,000; Full time

Job summary

Gold Rose AI seeks a software-focused product engineer to design and improve the agent tools, memory, and approval flows that control customer-facing AI actions. You will build evaluation datasets from anonymized owner conversations and validate every change through automated tests, ensuring accuracy and safe outputs.

You will reduce mistakes by anchoring actions to owner-approved prompts and memory rules, and track costs to keep the service affordable for small shops.

Qualifications

• 3+ years shipping software, with at least 1 year building on large language models in production.
• A habit of measuring quality instead of eyeballing it.
• Clear writing: prompts, specs and postmortems.

Responsibilities

• Build and improve agent tools, memory and approval flows.
• Create evaluation sets from real (anonymized) owner conversations and run them on every change.
• Reduce mistakes: wrong facts, made-up details, actions without permission.
• Track cost and speed so the product stays affordable for small shops.

Skills

Shipping software

Quality metrics

Clear writing

Job description

Gold Rose AI seeks a software-focused product engineer to design and improve the agent tools, memory, and approval flows that control customer-facing AI actions. You will build evaluation datasets from anonymized owner conversations and validate every change through automated tests, ensuring accuracy and safe outputs.

You will reduce mistakes by anchoring actions to owner-approved prompts and memory rules, and track costs to keep the service affordable for small shops.