Senior Applied Scientist, GenAI & Evaluation Leader (Remote)

Alcor; Medellín; Presencial; COP 378.310.000 - 472.888.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Life insurance

Funeral assistance

Home office stipend

100% remote work

Descripción de la vacante

Caseware is a global fintech leader offering accounting and audit software. This fully remote role is based in Colombia and focuses on building evaluation systems, synthetic data generation, and agent frameworks to improve quality and reliability of GenAI-powered products.

You will mentor teams and shape the technical direction while ensuring compliance with legal obligations. You will design experiments, own evaluation methodologies, and advance self-learning systems that improve with usage,

Formación

• 6+ years in applied science, machine learning, or data-intensive engineering.
• 2+ years in production GenAI/LLM-based systems.
• Experience designing evaluation frameworks for ML/LLM systems.
• Strong experimentation mindset and data-driven conclusions.
• Ability to deploy production tooling on AWS.

Responsabilidades

• Design and run experiments to measure and improve quality of LLM-based applications and agents.
• Build and own evaluation methodology used by internal teams and customers.
• Develop AI-based tools powering the end-to-end agent builder experience.
• Create synthetic data generation and eval-builder capabilities for agents.
• Define domain-rule based memory promotion/demotion criteria with stakeholders.
• Contribute to memory and self-learning improvements with safety and legal considerations.
• Mentor other scientists and engineers and influence technical direction.

Conocimientos

Applied science

GenAI

Experiment design

Production tooling

AWS

Statistics

English communication

Educación

MS or PhD in quantitative field

Herramientas

LangChain

LangSmith

LangGraph

AWS

OpenSearch

Descripción del empleo

Caseware is a global fintech leader offering accounting and audit software. This fully remote role is based in Colombia and focuses on building evaluation systems, synthetic data generation, and agent frameworks to improve quality and reliability of GenAI-powered products.

You will mentor teams and shape the technical direction while ensuring compliance with legal obligations. You will design experiments, own evaluation methodologies, and advance self-learning systems that improve with usage,