Senior Applied Scientist — AI Agents & Evaluation (Remote)

CaseWare; Bogotá; Presencial; COP 120.000.000 - 210.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote work

Home office stipend

Prepaid medicine

Life insurance

Internet allowance

Time off

Recognition rewards

Training budget

Descripción de la vacante

Caseware is seeking a senior to staff-level applied scientist to design experiments and build evaluation frameworks for GenAI-driven audit and accounting tools. You will own the evaluation methodology, develop AI-based tools for the agent builder, and create synthetic data generation capabilities.

You will translate domain procedures into robust verification structures and advance self-learning systems while upholding legal and contractual obligations.

Formación

• 6+ years (senior) to 8+ years (staff) of professional experience in applied science, machine learning, research, or data-intensive engineering.
• 2+ years on production GenAI or LLM-based systems (applications, agents, or tooling).
• Experience designing and building evaluation frameworks for ML/LLM systems: metrics, benchmarks, scoring approaches.
• Strong experimentation mindset with clear hypotheses and defensible conclusions from real data.
• Ability to build and operate production tooling and services, ideally on AWS.
• Understanding GenAI tradeoffs: quality, latency, cost, reliability, safety.

Responsabilidades

• Design and run experiments measuring and improving quality of LLM-based applications and agents.
• Build and own the evaluation methodology: benchmarks, regression suites, scoring, thresholds across task success, faithfulness, safety, latency, and cost.
• Develop AI-based tools powering the end-to-end agent builder experience used by Caseware teams and customers.
• Create synthetic data generation and eval-builder capabilities for internal teams and customers.
• Translate domain procedures into task-plus-verifier structures and analyze failure modes of LLM systems.
• Influence technical direction and mentor others as staff-level contributor.

Conocimientos

GenAI systems

Experimentation

Evaluation frameworks

AWS tooling

Statistical methods

English communication

Fast-paced environments

Memory/agent knowledge

Educación

PhD/MS in quantitative field (CS/Stats/ML)

Herramientas

AWS

LangChain

LangSmith

Descripción del empleo

Caseware is seeking a senior to staff-level applied scientist to design experiments and build evaluation frameworks for GenAI-driven audit and accounting tools. You will own the evaluation methodology, develop AI-based tools for the agent builder, and create synthetic data generation capabilities.

You will translate domain procedures into robust verification structures and advance self-learning systems while upholding legal and contractual obligations.