Senior AI Engineer - Agent Quality & Evaluations (Remote)

vibehackers; Northern (KY); Hybrid; USD 150,000 - 250,000; Full time

Benefits offered by this job

Health insurance

Parental leave

Unlimited flexible time off

401(k) match

Learning stipend

Job summary

Flodesk is seeking a Senior Applied AI Engineer to own agent quality and evaluation for member-facing AI features. You will build prompting, agent instructions, evaluation infrastructure, monitoring, and tooling needed to measure and improve production LLM/agent behavior across the product.

Responsibilities include defining how context, member data, tools, and product state should be used; prototyping agent loops, tools, and structured interfaces; and building scalable evaluation harnesses with

Qualifications

• Proven track record building or meaningfully improving production LLM/agentic products.
• Strong software engineering skills in Python, TypeScript, or similar languages to build internal tools, evaluation systems, and prototypes.
• Clear understanding of interactions among prompts, context, tools, agent loops, orchestration, and product state.

Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs (segmentation, scheduling, analytics, recommendations).
• Create, test, version, and document prompts and agent instructions with a clear change history and measurable impact.
• Define how context, member data, tools, and product state should be used; prototype agent loops, tools, and structured interfaces.
• Own AI evaluation practice: representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.
• Build and operate evaluation harnesses, monitoring, and quality dashboards to enable scalable measurement.
• Diagnose failures across prompts, context, orchestration, models, tools, data, and product code; fix issues or partner with owners.
• Set quality baselines and release gates for prompt, model, and agent changes; convert production failures and feedback into regression cases.
• Partner with product, design, marketing, and copy experts to encode product point-of-view while contributing product judgment.

Skills

Systems thinking

Software engineering

Prompt engineering

Evaluation & observability

Experimental design

Monitoring and diagnostics

Product sense

Prototyping

Cross-functional collaboration

Documentation and versioning

Quality assurance

Tools

Python

TypeScript

Langfuse

Braintrust

Observability platforms

Job description

Flodesk is seeking a Senior Applied AI Engineer to own agent quality and evaluation for member-facing AI features. You will build prompting, agent instructions, evaluation infrastructure, monitoring, and tooling needed to measure and improve production LLM/agent behavior across the product.

Responsibilities include defining how context, member data, tools, and product state should be used; prototyping agent loops, tools, and structured interfaces; and building scalable evaluation harnesses with