AI Scientist & Evaluation Engineer — Banking AI

Hub; København; On-site; DKK 900,000 - 1,200,000; Full time

Job summary

Shine is building a Banking AI Foundation & Ops Efficiency team to scale AI-driven workflows with rigorous evaluation and compliant pipelines. We seek a Data Scientist to define evaluation metrics, curate golden datasets, and shape how models are tested across banking use cases.

You'll collaborate with software engineers to ship end-to-end systems, own privacy safeguards, and implement observability with Langfuse to keep agent behaviour transparent and auditable.

Qualifications

• Define and implement evaluation metrics tailored to what each use case needs to get right.
• Curate and maintain rom 'golden' datasets to compare real behaviour against the ideal result.
• Analyse historical data to determine what a given system's evals should be checking for in the first place.
• Monitor the AI/LLM-based components of our systems in production to catch regressions before they reach customers or compliance reviewers.
• Help build the agentic systems themselves, from single-purpose LLM agents to multi-agent systems with tool-calling and orchestration.
• Own privacy and compliance safeguards, ensuring sensitive data gets detected and redacted correctly and that every system is auditable end to end.
• Build and maintain observability for our agentic components using an LLM observability tool (Langfuse) to ensure agent behaviour stays traceable and transparent.
• Partner with the team's software engineers to take the project to production.
• Contribute to prioritising the automation opportunities across Banking that are worth pursuing, and help other teams adopt what we build.

Responsibilities

• Define and implement evaluation metrics for our AI systems, tailored to what each use case needs to get right.
• Curate and maintain golden datasets to compare a system's real behaviour against the ideal result.
• Analyse historical data to determine what a given system's evals should be checking for in the first place.
• Monitor the AI/LLM-based components of our systems in production to catch regressions before they reach customers or compliance reviewers.
• Help build the agentic systems themselves, from single-purpose LLM agents to multi-agent systems with tool-calling and orchestration.
• Own privacy and compliance safeguards, ensuring sensitive data gets detected and redacted correctly and that every system is auditable end to end.
• Build and maintain observability for our agentic components using an LLM observability tool (Langfuse) to ensure agent behaviour stays traceable and transparent.
• Partner with the team's software engineers to take the project to production.
• Contribute to prioritising the automation opportunities across Banking that are worth pursuing, and help other teams adopt what we build.

Skills

Evaluation metrics

Golden datasets

Historical data analysis

Production monitoring

Agentic systems

Privacy & compliance

Observability (Langfuse)

Collaboration with engineers

Automation prioritisation

Tools

Langfuse

Job description

Shine is building a Banking AI Foundation & Ops Efficiency team to scale AI-driven workflows with rigorous evaluation and compliant pipelines. We seek a Data Scientist to define evaluation metrics, curate golden datasets, and shape how models are tested across banking use cases.

You'll collaborate with software engineers to ship end-to-end systems, own privacy safeguards, and implement observability with Langfuse to keep agent behaviour transparent and auditable.