Senior AI Scientist: Evaluation & Agentic Memory

Caseware; Medellín; Presencial; COP 264.197.000 - 372.984.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote work

Prepaid medicine

Life insurance

Internet allowance

Home office stipend

Competitive compensation

100% remote work environment

Flexible time-off

Descripción de la vacante

Caseware is seeking a senior to staff level Applied Scientist to design and run experiments, own evaluation frameworks, and build AI tools for an end-to-end agent builder platform. You will craft synthetic data and eval builders, advance self-learning systems, and ensure legal and contractual obligations are respected as memory thresholds are defined and promoted.

Remote role based in Colombia, collaborating with global teams to push the boundaries of audit and financial reporting tooling using

Formación

• 6+ years in applied science, ML, research, or data-intensive engineering. Staff-level candidates bring impact beyond a single team.
• 2+ years working on production GenAI or LLM-based systems (applications, agents, or the tooling and evaluation around them).
• Proven experience designing and building evaluation frameworks for ML/LLM systems: metrics, benchmarks, scoring approaches, and rigorous experiment design.
• Strong experimentation mindset with clear hypotheses and defensible conclusions from real data.

Responsabilidades

• Design and run experiments that measure and improve quality of LLM-based applications and agents.
• Build and own the evaluation methodology used by internal teams and customers: benchmarks, regression suites, scoring, thresholds, and drift checks.
• Develop AI-based tools and capabilities powering the end-to-end agent builder experience for Caseware teams and customers.
• Create synthetic data generation and eval-builder capabilities for agents.
• Translate domain procedures into task-plus-verifier structures and study LLM failure modes.

Conocimientos

Applied science

Machine learning

Experiment design

LLM evaluation

Production tooling

AWS

English communication

Mentoring

Educación

PhD or MS in quantitative field

Herramientas

AWS

LangFuse

LangSmith

LangChain

Descripción del empleo

Caseware is seeking a senior to staff level Applied Scientist to design and run experiments, own evaluation frameworks, and build AI tools for an end-to-end agent builder platform. You will craft synthetic data and eval builders, advance self-learning systems, and ensure legal and contractual obligations are respected as memory thresholds are defined and promoted.

Remote role based in Colombia, collaborating with global teams to push the boundaries of audit and financial reporting tooling using