Data Scientist - AI Engineer

Shine; Berlin; On-site; EUR 75,000 - 120,000; Full time

Job summary

Shine is hiring a Data Scientist to contribute to the evaluation process of our AI-driven banking solutions. You will help design evaluation pipelines, curate datasets, and ensure privacy and compliance throughout development and production.

You will collaborate with software engineers to deploy end-to-end agentic systems, from single-purpose LLM agents to multi-agent orchestrations, while maintaining observability and auditable workflows.

Qualifications

• Experience defining and implementing evaluation metrics for AI systems.
• Ability to curate and maintain golden datasets for benchmarking AI behaviour.
• Skill in analyzing historical data to define relevant eval checks.
• Experience monitoring AI/LLM components in production for regressions.

Responsibilities

• Define and implement evaluation metrics for AI systems, tailored to each use case.
• Curate golden datasets to compare real vs ideal behaviour.
• Analyze historical data to set eval priorities.
• Monitor AI components in production to catch regressions.
• Help build agentic systems from single to multi-agent orchestration.

Skills

Evaluation metrics

Golden datasets

Privacy safeguards

Agentic systems

Tools

Langfuse

Job description

About Shine

Shine is the financial copilot for entrepreneurs and small business owners. Founded by serial entrepreneurs Rico Andersen and Martin Hegelund, Shine is a leading European fintech unicorn on a mission to restore the joy of running a business, by ending wasted time on financial admin. Shine offers a connected solution for invoicing, accounting, payroll, business accounts, payments, and financing, meaning business owners can focus their energy on growing a healthy business, not held back by manual admin. Part of something bigger Today we're part of Cegid, a European leader in cloud software for finance and accounting. Together we're building Europe's leading financial copilot for small businesses and their accountants. Shine already supports more than 400,000 small businesses. As part of Cegid, we now reach over one million small businesses and 15,000 accountants across Europe. We're a multicultural team working from France, Germany, Denmark and the Netherlands, contributing to a wider European network that spans Spain, Portugal. Your hiring experience matters Just as we respect our customers' time, we respect yours. Your experience with Shine and Cegid should feel simple, transparent and genuinely supportive.

The Banking AI Foundation & Ops Efficiency Team at Shine.

Our Banking AI Foundation & Ops Efficiency team accelerates operational efficiency across the Banking Domain by identifying and scaling high-impact, compliant AI-driven workflows.

We're a cross-functional team of Software/AI engineers and data scientists who build these systems end-to-end together.

Our work covers a wide range of AI-driven solutions depending on each team’s specific needs and goals.

Evaluation, explainability, and traceability of these AI systems are a dedicated focus within our team, and the data scientists own that foundation across every project we ship.

Your role as a Data Scientist

We're looking for a Data Scientist/AI Engineer to contribute to the evaluation process of our system, as well as contribute to the orchestration design and AI safety foundation behind our AI-driven workflows.

AI solutions are scaling quickly across the Banking Domain, and each new one needs the same evaluation rigour and compliance safeguards we've already built into our existing systems, applied consistently rather than reinvented from scratch each time.

You will collaborate with the team to build and manage agentic systems, and be a key driver of the design of the evaluation process for these systems, with privacy and compliance safeguards in mind throughout the whole process. You will work together with your team to ship end-to-end systems.

About you

• Define and implement evaluation metrics for our AI systems, tailored to what each use case needs to get right.

• Curate and maintain \"golden\" datasets to compare a system's real behaviour against the ideal result.

• Analyse historical data to determine what a given system's evals should be checking for in the first place.

• Monitor the AI/LLM-based components of our systems in production to catch regressions before they reach customers or compliance reviewers.

• Help build the agentic systems themselves, from single-purpose LLM agents to multi-agent systems with tool-calling and orchestration.

• Own privacy and compliance safeguards, ensuring sensitive data gets detected and redacted correctly and that every system is auditable end to end.

• Build and maintain observability for our agentic components using an LLM observability tool (Langfuse) to ensure agent behaviour stays traceable and transparent.

• Partner with the team's software engineers to take the project to production.

• Contribute to prioritising the automation opportunities across Banking that are worth pursuing, and help other teams adopt what we build.

What success looks like

90 Days: Onboarded on the team's tooling and one live project. Shipped a first evaluation metric or golden dataset, and/or contributed to building or debugging an agent, with close support. Started a well-scoped task end to end. Relationships built with the team's engineers and senior teammates.

6 Months: Owns an evaluation pipeline and/or an agent-building or orchestration workstream for a live project end to end. Production issues within their scope are fixed independently rather than routed elsewhere. Design and evaluation trade-offs explained clearly to the team being supported. Contributing to a second project.

12 Months: Takes a loosely-specified problem, whether that's a new agent to build, a system to orchestrate, or a new eval strategy to define, from data analysis through implementation independently. Trusted as a second opinion on eval design and/or agent architecture by teammates. Contributing to prioritisation or adoption conversations across Banking and CX. Able to directly support a newer joiner on the team.

Equal Opportunity Employer

We follow the principle of equal treatment to consider all job applicants and do not discriminate based on their gender, sexual orientation, color, racial or ethnic origin, religion, disability, etc. as per applicable law.