Senior Applied AI Engineer - Agent Quality Evaluations

Flodesk; San Francisco (CA); A distancia; USD 165.000 - 250.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Fully paid health insurance

16 weeks paid parental leave

22 weeks paid maternity leave

Unlimited flexible time off

401(k) match

Learning stipend

Descripción de la vacante

Flodesk is a remote-first company with teams across the globe, headquartered in San Francisco. We’re building AI-driven marketing tools to help small businesses scale, with hub offices and a distributed workforce.

You will shape the AI system, prompts, and evaluation to deliver reliable, scalable experiences for users. Join Flodesk to define how our system uses context, data and product state, while partnering with design and product to align with user goals and maintain high quality across

Formación

• Experience building or significantly improving production LLM/agentic products.
• Strong software engineering skills in Python/TypeScript and internal toolwork.
• Experience with evaluation, observability and design of experiments for AI systems.

Responsabilidades

• Manage member-facing AI behavior across prompts, edits and future AI capabilities.
• Create, test, version and document prompts and agent instructions with traceability.
• Define system use of context, data, tools and product state; prototype agent loops and interfaces.
• Own AI evaluation practices with datasets, scenarios, rubrics and regression suites.
• Build eval harnesses and dashboards to measure quality at scale.
• Diagnose failures across prompts, context, orchestration, models and data; fix root causes.
• Set quality baselines and release gates for AI changes; transform feedback into regression cases.
• Collaborate with product, design, marketing and copy teams to encode product vision.

Conocimientos

Production LLM

Python

TypeScript

Prompt engineering

Observability

Herramientas

Langfuse

Braintrust

Descripción del empleo

Flodesk is a remote-first company with teams across the globe, headquartered in San Francisco. We’re building AI-driven marketing tools to help small businesses scale, with hub offices and a distributed workforce.

You will shape the AI system, prompts, and evaluation to deliver reliable, scalable experiences for users. Join Flodesk to define how our system uses context, data and product state, while partnering with design and product to align with user goals and maintain high quality across