Senior AI Systems Engineer, Agent Quality & Evaluation

Flodesk; Los Angeles (CA); Remote; USD 150,000 - 250,000; Full time

Benefits offered by this job

Health insurance

Parental leave

Unlimited time off

401(k) match

Learning stipend

Job summary

Flodesk is a remote-first company shaping the future of AI-driven marketing tools for small businesses. The role focuses on owning prompts, agent instructions, evaluation criteria, and the continuous-improvement loop across prompts, context, tools and product state, ensuring measurable quality.

You will collaborate with product, design, marketing and copy teams to encode Flodesk's perspective while applying your judgment on how the product should behave, balancing quality, latency and cost.

Qualifications

• Production LLM or agentic product experience.
• Strong software engineering in Python/TypeScript.
• Understanding prompts, context, tools, and product state.
• Experience in evaluation design and observability.
• Translate feedback into system changes.
• Clear communication across disciplines.

Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing and future jobs such as segmentation, scheduling, analytics and recommendations.
• Create, test, version and document prompts and agent instructions, with a clear record of what changed, why it changed and how behavior improved.
• Define how the system should use context, member data, tools and product state. Build and prototype the agent loops, tools and structured interfaces needed to make those behaviors work.
• Own Flodesk's AI evaluation practice, including representative datasets, behavioral scenarios, scoring rubrics, regression suites and human review.
• Build and operate eval harnesses, monitoring and quality dashboards so the team can measure quality without relying on one-off manual checks.
• Diagnose failures across prompts, context, orchestration, models, tools, data and product code, then fix the right layer or partner with the engineer who owns it.
• Set quality baselines and release gates for prompt, model and agent changes. Turn production failures, traces and member feedback into durable regression cases.
• Partner with product, design, marketing and copy experts to encode Flodesk's point of view into the system, while bringing your own judgment about how the product should behave.

Skills

LLM product experience

Python

TypeScript

Systems thinking

Cross-functional collaboration

Observability

Tools

Langfuse

Braintrust

Evaluation tooling

Job description

Flodesk is a remote-first company shaping the future of AI-driven marketing tools for small businesses. The role focuses on owning prompts, agent instructions, evaluation criteria, and the continuous-improvement loop across prompts, context, tools and product state, ensuring measurable quality.

You will collaborate with product, design, marketing and copy teams to encode Flodesk's perspective while applying your judgment on how the product should behave, balancing quality, latency and cost.