Senior AI Systems Engineer — Agent Quality & Evaluation

Flodesk; Seattle (WA); A distancia; USD 150.000 - 250.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Health insurance (fully paid for indiv

Parental leave (16–22 weeks)

Unlimited flexible time off

401(k) match (US employees only)

Learning & development stipend

Descripción de la vacante

Flodesk, a remote-first email marketing company, seeks an experienced AI Systems professional to own prompts, agent instructions, quality definitions, and evaluation infrastructure. You’ll connect engineering, product, and language models to improve member experiences and build scalable tooling for continuous improvement.

You will partner with product, design, marketing and copy teams to embed Flodesk's perspective into the system while ensuring reliable, measurable performance across prompts,

Formación

• A track record of building or meaningfully improving production LLM or agentic products.
• Strong software engineering skills in Python and TypeScript.
• Understanding of prompts, context, tools, and product state.
• Experience with LLM evaluation, observability and experimental design.
• Ability to translate ambiguous feedback into testable quality definitions.

Responsabilidades

• Manage member-facing AI behavior across prompt-to-create and future jobs.
• Create, test, version and document prompts and agent instructions.
• Define how the system uses context, data, tools and product state.
• Own AI evaluation practices, datasets, rubrics and regression suites.
• Build eval harnesses and dashboards to measure quality at scale.
• Diagnose failures across prompts, orchestration, models and data.
• Set quality baselines and release gates for changes.
• Collaborate with product, design, marketing and copy teams to encode the company perspective.

Conocimientos

LLM/Agentic product development

Python

TypeScript

Prompts/Context/Tools/Agent loops

LLM evaluation

Observability

Experimental design

Product sense

Systems thinking

Cross-functional communication

Herramientas

Langfuse

Braintrust

Descripción del empleo

Flodesk, a remote-first email marketing company, seeks an experienced AI Systems professional to own prompts, agent instructions, quality definitions, and evaluation infrastructure. You’ll connect engineering, product, and language models to improve member experiences and build scalable tooling for continuous improvement.

You will partner with product, design, marketing and copy teams to embed Flodesk's perspective into the system while ensuring reliable, measurable performance across prompts,