Senior Applied AI Engineer, Agent Quality & Evaluations

Flodesk; San Francisco (CA); Remote; USD 165,000 - 250,000; Full time

Benefits offered by this job

Fully paid health insurance

16 weeks paid parental leave

22 weeks paid maternity leave

Unlimited flexible time off

401(k) match

Learning stipend

Job summary

Flodesk is a remote-first company with teams across the globe, headquartered in San Francisco. We’re building AI-driven marketing tools to help small businesses scale, with hub offices and a distributed workforce.

You will shape the AI system, prompts, and evaluation to deliver reliable, scalable experiences for users. Join Flodesk to define how our system uses context, data and product state, while partnering with design and product to align with user goals and maintain high quality across

Qualifications

• Experience building or significantly improving production LLM/agentic products.
• Strong software engineering skills in Python/TypeScript and internal toolwork.
• Experience with evaluation, observability and design of experiments for AI systems.

Responsibilities

• Manage member-facing AI behavior across prompts, edits and future AI capabilities.
• Create, test, version and document prompts and agent instructions with traceability.
• Define system use of context, data, tools and product state; prototype agent loops and interfaces.
• Own AI evaluation practices with datasets, scenarios, rubrics and regression suites.
• Build eval harnesses and dashboards to measure quality at scale.
• Diagnose failures across prompts, context, orchestration, models and data; fix root causes.
• Set quality baselines and release gates for AI changes; transform feedback into regression cases.
• Collaborate with product, design, marketing and copy teams to encode product vision.

Skills

Production LLM

Python

TypeScript

Prompt engineering

Observability

Tools

Langfuse

Braintrust

Job description

Flodesk is recognized in the Inc 5000 as one of the world's fastest-growing email marketing companies, built to help entrepreneurs sell online and design emails that people love to get. We're committed to giving small businesses simple and intuitive tools that help them grow, nurture, and monetize their email list.

We’re a remote-first company headquartered in San Francisco with a globally distributed team, including in-person hubs in Da Nang (Vietnam), Barcelona (Spain), and Menlo Park (California). Our team reflects the diversity and creativity of the people we serve. Join our mission to level the playing field for small business owners through good design.

About The Role

Flodesk is building toward a future where small business owners do not need to be expert marketers to grow. Today, our AI helps members create and edit emails, but that is only the starting point. We are building a system that can understand a member's business, brand, audience and past performance, then help turn a goal into a complete marketing campaign across email, workflows, forms, sales pages, segmentation, scheduling, analytics and recommendations.

What you'll do

• Manage member-facing AI behavior across prompt-to-create, agentic editing and future jobs such as segmentation, scheduling, analytics and recommendations.
• Create, test, version and document prompts and agent instructions, with a clear record of what changed, why it changed and how behavior improved
• Define how the system should use context, member data, tools and product state. Build and prototype the agent loops, tools and structured interfaces needed to make those behaviors work
• Own Flodesk's AI evaluation practice, including representative datasets, behavioral scenarios, scoring rubrics, regression suites and human review
• Build and operate eval harnesses, monitoring and quality dashboards so the team can measure quality without relying on one-off manual checks
• Diagnose failures across prompts, context, orchestration, models, tools, data and product code, then fix the right layer or partner with the engineer who owns it.
• Set quality baselines and release gates for prompt, model and agent changes. Turn production failures, traces and member feedback into durable regression cases
• Partner with product, design, marketing and copy experts to encode Flodesk's point of view into the system, while bringing your own judgment about how the product should behave

What you'll bring

• A track record of building or meaningfully improving production LLM or agentic products, not just prototypes
• Strong software engineering skills in Python, TypeScript or a similar language that let you build reliable internal tools, evaluation systems and prototypes
• The interaction between prompts, context, tools, agent loops, orchestration and product state is clear to you
• LLM evaluation, observability and experimental design are areas where you have direct experience, and you know how to combine automated checks with human judgment
• A strong product sense that helps you translate ambiguous member feedback into testable quality definitions and concrete system changes
• Systems thinking and reusable patterns guide your work, so today's email experience gets better without building in assumptions that make tomorrow's surfaces harder to build
• Across technical and nontechnical disciplines, your communication is clear and you can explain tradeoffs involving quality, latency, reliability and cost

Bonus points if you've

• Built content-generation, creative or marketing tools
• Built multi-turn agents that use tools and preserve user intent across edits
• Used Langfuse, Braintrust or comparable evaluation and observability platforms
• Worked on products for small businesses, creators or marketers

What we bring

• $150,000 - $250,000 base salary, depending on your location and experience
  • Tier 1 city residents: $165,000 - $250,000
  • All other US based locations: $150,000 - $235,000
• Fully paid health insurance for individual coverage
• 16 weeks paid parental leave for non-birthing parents; 22 weeks paid maternity leave for birthing parents
• Unlimited flexible time off
• 401(k) match (US employees only)
• $1,000 annual stipend for learning and development

Flodesk is an equal opportunity employer. We are committed to building a diverse and inclusive workforce and do not discriminate based on race, religion, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, veteran or military status, or any other legally protected characteristics, Flodesk is committed to providing reasonable accommodations for candidates with disabilities who need assistance during the hiring process. To request a reasonable accommodation, please email [e-mail]

Notice to California-based Candidates and Vietnam-based Candidates for Employment. This Candidate Privacy Notice is intended to provide information about how Flodesk collects and uses personal information to California consumers and candidates located in Vietnam (Vietnam-based candidates) who apply for employment with Flodesk. If you are employed by Flodesk, refer to the Employee Handbook for additional information. For any questions about this notice, please contact [e-mail]
Personal Information Flodesk Collects:
Identifiers Including name, address, email, telephone number, social security number, driver license number, passport number, and other personal identifying information. For California-based candidates: Characteristics of protected classifications under California or federal law including demographic information and other personal information obtained during the application process, such as gender, race, national origin. Professional or employment-related information, such as salary/compensation and benefits packages, other relocation or job preferences, prior background, experience, skills, and other information in support of your application, reference information. Any other information you provide as a part of recruitment, job application, or interview process.
Purposes for Collecting Personal Information:
To consider qualifications, skills, and interest for employment. To communicate with you during the recruitment and interview process. To provide compensation, including payroll, and administer stock options and benefits, including medical, dental, vision, commuter, and retirement benefits. To provide human resources services and conduct performance evaluations. To monitor work eligibility including work-related licenses, credentials, training, and eligibility to work in the United States or in Vietnam. To improve recruitment and interview processes and ensure a safe and efficient working environment. To comply with applicable legal or regulatory requirements as Flodesk may transfer or store internationally your information, including to or in the United States, European Union and Vietnam and in the cloud, and this data may be subject to the laws and accessible to the courts, law enforcement and national security authorities of such jurisdictions.
For Vietnam-based candidates:
By clicking "Submit Application", You confirm that you have read the Privacy Notice and agree to allow Flodesk to process your personal data for this application, including cross-border transfer. You have the right to withdraw your consent or request data deletion at any time by contacting [e-mail]