Senior Applied AI Engineer, Agent Quality & Evaluations

vibehackers; Northern (KY); Hybrid; USD 150,000 - 250,000; Full time

Benefits offered by this job

Health insurance

Parental leave

Unlimited flexible time off

401(k) match

Learning stipend

Job summary

Flodesk is seeking a Senior Applied AI Engineer to own agent quality and evaluation for member-facing AI features. You will build prompting, agent instructions, evaluation infrastructure, monitoring, and tooling needed to measure and improve production LLM/agent behavior across the product.

Responsibilities include defining how context, member data, tools, and product state should be used; prototyping agent loops, tools, and structured interfaces; and building scalable evaluation harnesses with

Qualifications

• Proven track record building or meaningfully improving production LLM/agentic products.
• Strong software engineering skills in Python, TypeScript, or similar languages to build internal tools, evaluation systems, and prototypes.
• Clear understanding of interactions among prompts, context, tools, agent loops, orchestration, and product state.

Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs (segmentation, scheduling, analytics, recommendations).
• Create, test, version, and document prompts and agent instructions with a clear change history and measurable impact.
• Define how context, member data, tools, and product state should be used; prototype agent loops, tools, and structured interfaces.
• Own AI evaluation practice: representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.
• Build and operate evaluation harnesses, monitoring, and quality dashboards to enable scalable measurement.
• Diagnose failures across prompts, context, orchestration, models, tools, data, and product code; fix issues or partner with owners.
• Set quality baselines and release gates for prompt, model, and agent changes; convert production failures and feedback into regression cases.
• Partner with product, design, marketing, and copy experts to encode product point-of-view while contributing product judgment.

Skills

Systems thinking

Software engineering

Prompt engineering

Evaluation & observability

Experimental design

Monitoring and diagnostics

Product sense

Prototyping

Cross-functional collaboration

Documentation and versioning

Quality assurance

Tools

Python

TypeScript

Langfuse

Braintrust

Observability platforms

Job description

Senior Applied AI Engineer, Agent Quality & Evaluations

Building AI dev tools and evaluation for LLMs and agents; focuses on prompts, agent loops, and observability.

About the Role

Flodesk is hiring a Senior Applied AI Engineer to own agent quality and evaluation for member-facing AI features. You will build the prompting, agent instructions, evaluation infrastructure, monitoring, and tooling needed to measure and improve production LLM/agent behavior across the product.

Job Description

Role

Flodesk is seeking a Senior Applied AI Engineer (Agent Quality & Evaluations) to report to the Head of Product, AI Systems and Core Experience. The role focuses on shaping AI-driven member experiences by owning prompts, agent instructions, quality definitions, evaluation criteria, and the continuous-improvement loop through technical tooling and evaluation infrastructure.

Key Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs (segmentation, scheduling, analytics, recommendations).
• Create, test, version, and document prompts and agent instructions with a clear change history and measurable impact.
• Define how context, member data, tools, and product state should be used; prototype agent loops, tools, and structured interfaces.
• Own AI evaluation practice: representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.
• Build and operate evaluation harnesses, monitoring, and quality dashboards to enable scalable measurement.
• Diagnose failures across prompts, context, orchestration, models, tools, data, and product code; fix issues or partner with owners.
• Set quality baselines and release gates for prompt, model, and agent changes; convert production failures and feedback into regression cases.
• Partner with product, design, marketing, and copy experts to encode product point-of-view while contributing product judgment.

Requirements

• Proven track record of building or meaningfully improving production LLM or agentic products (not just prototypes).
• Strong software engineering skills in Python, TypeScript, or similar languages to build internal tools, evaluation systems, and prototypes.
• Clear understanding of interactions among prompts, context, tools, agent loops, orchestration, and product state.
• Direct experience with LLM evaluation, observability, and experimental design; ability to combine automated checks with human judgment.
• Strong product sense to translate ambiguous member feedback into testable quality definitions and concrete system changes.
• Systems thinking and emphasis on reusable patterns to avoid short-term assumptions that hinder future surfaces.
• Clear communication across technical and nontechnical disciplines; able to explain tradeoffs involving quality, latency, reliability, and cost.

Bonus / Nice to have

• Experience building content-generation, creative, or marketing tools.
• Experience building multi-turn agents that use tools and preserve user intent across edits.
• Experience with Langfuse, Braintrust, or comparable evaluation and observability platforms.
• Experience working on products for small businesses, creators, or marketers.
• Base salary range: $150,000 - $250,000 (tiers noted: Tier 1 cities $165,000 - $250,000; other US locations $150,000 - $235,000).
• Fully paid health insurance for individual coverage.
• Parental leave: 16 weeks paid for non-birthing parents; 22 weeks paid maternity leave for birthing parents.
• Unlimited flexible time off.
• 401(k) match (US employees only).
• $1,000 annual stipend for learning and development.
• Remote-first company with globally distributed team and in-person hubs.

Python TypeScript LLMs Langfuse Braintrust

Skills

Systems thinking Software engineering Prompt engineering Evaluation & observability Experimental design Monitoring and diagnostics Product sense Prototyping Cross-functional collaboration Documentation and versioning Quality assurance

Experience Level

Senior

USD 150,000 - 250,000/year

Employment Type

Full Time

• Fully paid health insurance (individual coverage)
• 16 weeks paid parental leave (non-birthing parents)
• 22 weeks paid maternity leave (birthing parents)
• Unlimited flexible time off
• 401(k) match (US employees only)
• $1,000 annual learning and development stipend