AI Engineer - Evals

BlueCrest Capital Management

Greater London

Presencial

GBP 90.000 - 130.000

Jornada completa

Hace 2 días
Sé de los primeros/as/es en solicitar esta vacante

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

BlueCrest Capital Management is seeking an AI Engineer focused on evals to build and govern AI tooling for front office teams. You will extend evaluation frameworks, assess tool-use and agent behavior, and target measurable improvements across coding assistants, RAG pipelines and governance processes.

In a fast-moving environment, you will help set direction for AI evaluation, educate engineers, and ensure evidence-based decision making across the firm’s AI initiatives.

Formación

  • 5+ years in software engineering with strong Python and SQL.
  • Hands-on experience evaluating LLM systems: LLM-as-judge, human-labelled datasets, retrieval metrics and agent trace analysis.
  • Hands-on experience with eval and observability tooling (e.g. Azure AI Foundry evals, Langfuse, LangSmith, MLflow, promptfoo, Inspect).
  • Strong testing discipline and an experimental mindset: baselines, controls, significance and honest reporting of results.
  • Experience building LLM applications: retrieval (RAG), tool use and agents.
  • Heavy daily use of agentic coding tools, with a clear view of where they fail and how to measure them.
  • A track record of shipping production systems: APIs, containers, CI/CD and cloud (Azure or Kubernetes preferred).
  • The ability to explain evaluation concepts and results clearly, and to bring teams with you.

Responsabilidades

  • Extend our evaluation framework to cover new assistants, agents and use cases, improving its design, reliability and ease of use.
  • Set the direction for AI evaluation, balancing new capability against keeping the framework simple to adopt.
  • Educate engineers and the business on why evals matter, through documentation, workshops, pairing and code review.
  • Help teams define what good looks like for their use cases and turn it into measurable criteria.
  • Add evaluation of RAG, agents and multi-step tool use: tool-call correctness, task completion, groundedness and citation accuracy.
  • Evaluate agent behaviour end to end through traces, not just final answers.
  • Evaluate AI coding tools on the firm’s own codebases and tasks.
  • Measure how effective these tools are in practice: task success, code quality, review effort and delivery speed.
  • Turn results into practical guidance on tool choice, configuration and ways of working across engineering.
  • Build and maintain ground-truth datasets with PMs, quants and researchers.
  • Keep datasets versioned, governed and reusable across teams.
  • Run experiment tracking and integrate evals into CI/CD, so model, prompt and retrieval changes are gated on measured quality.
  • Make the framework easy for other engineers to adopt by default.
  • Support red-teaming, failure analysis and the evidence needed for governance approval before deployment.
  • Monitor quality in production and catch regressions early.
  • Pick up other priorities across the team as they arise, from new front office tools to knowledge services, MCP integrations and data access.
  • Contribute to shared engineering standards, code review and production support.
  • Measure performance, accuracy and cost together, giving clear evidence for model, provider and adoption decisions.

Conocimientos

Python
SQL
LLM evals
Agent tracing
Experimentation
CI/CD

Herramientas

Azure Foundry evals
Langfuse
LangSmith
MLflow
promptfoo
Inspect

Descripción del empleo

The Team

The AI Team, part of Front Office Technology, builds the AI tools and platforms used across the front office: assistants and agents for investment teams, secure access to firm data and knowledge, and the controls that let AI be used safely in a regulated trading environment.

Role Purpose

We build central capabilities for the whole firm and support teams across the business as they build their own, bringing the most successful into Technology ownership as they grow in importance.

The Team

The AI Team, part of Front Office Technology, builds the AI tools and platforms used across the front office: assistants and agents for investment teams, secure access to firm data and knowledge, and the controls that let AI be used safely in a regulated trading environment.

The Role

As an AI Engineer with a focus on evals, you will help build the AI tools and platforms used across the front office, and the evidence that tells us whether they are accurate, safe and worth the spend.

AI-assisted development is becoming the default for our engineers, and assistants and agents are taking on more work in front office workflows. Decisions about which models, tools and changes to ship should rest on measurement, not impressions.

You will build on our existing evaluation framework rather than start from scratch, and help make good evaluation practice a normal part of how AI is built across the firm.

The team is small and priorities shift as the firm’s use of AI grows. You will be expected to pick up other priorities across the team as they arise, so this role will not be dedicated solely to evals.

Key Responsibilites
Evolving our evaluation framework
  • Extend our evaluation framework to cover new assistants, agents and use cases, improving its design, reliability and ease of use.
  • Set the direction for AI evaluation, balancing new capability against keeping the framework simple to adopt.
Education and advocacy
  • Educate engineers and the business on why evals matter, through documentation, workshops, pairing and code review.
  • Help teams define what good looks like for their use cases and turn it into measurable criteria.
Agentic and RAG evals
  • Add evaluation of RAG, agents and multi-step tool use: tool-call correctness, task completion, groundedness and citation accuracy.
  • Evaluate agent behaviour end to end through traces, not just final answers.
Coding tool effectiveness
  • Evaluate AI coding tools (e.g. Claude Code, Codex, GitHub Copilot) on the firm’s own codebases and tasks.
  • Measure how effective these tools are in practice: task success, code quality, review effort and delivery speed, and where they help or hinder.
  • Turn results into practical guidance on tool choice, configuration and ways of working across engineering.
Ground truth and datasets
  • Build and maintain ground-truth datasets with PMs, quants and researchers.
  • Keep datasets versioned, governed and reusable across teams.
Evals in the delivery pipeline
  • Run experiment tracking and integrate evals into CI/CD, so model, prompt and retrieval changes are gated on measured quality.
  • Make the framework easy for other engineers to adopt by default.
Assurance and governance
  • Support red-teaming, failure analysis and the evidence needed for governance approval before deployment.
  • Monitor quality in production and catch regressions early.
Wider AI engineering
  • Pick up other priorities across the team as they arise, from new front office tools to knowledge services, MCP integrations and data access.
  • Contribute to shared engineering standards, code review and production support.
Business outcomes
  • Measure performance, accuracy and cost together, giving clear evidence for model, provider and adoption decisions.
What Success Looks Like
First 6 months
  • Evaluation framework extended to cover the firm’s core AI assistants.
  • First coding assistant evaluation on firm codebases, with recommendations adopted.
  • Eval guidance and training in place for engineers building AI tools.
First 12 months
  • Evals gating releases in CI/CD for the firm’s AI assistants and agents.
  • Eval tooling used by engineers across the firm by default.
  • Coding tool effectiveness measured routinely, informing tooling and ways of working.
Key Competencies

You are a senior engineer with a scientist’s instinct. You want to know whether something actually works, and you design tests that would show you if it didn’t.

You understand how LLMs fail, and you are rigorous about what a result does and does not prove.

You turn complex results into clear recommendations for engineers, governance forums and senior stakeholders, and you are comfortable saying when a tool or model is not good enough.

You enjoy teaching others why evals matter, and are happy to switch to whatever matters most to the team when priorities change.

Essential
Key Experience and Qualifications
  • 5+ years in software engineering, with strong Python and SQL.
  • Hands‑on experience evaluating LLM systems: LLM-as-judge, human‑labelled datasets, retrieval metrics and agent trace analysis.
  • Hands‑on experience with eval and observability tooling (e.g. Azure AI Foundry evals, Langfuse, LangSmith, MLflow, promptfoo, Inspect).
  • Strong testing discipline and an experimental mindset: baselines, controls, significance and honest reporting of results.
  • Experience building LLM applications: retrieval (RAG), tool use and agents.
  • Heavy daily use of agentic coding tools, with a clear view of where they fail and how to measure them.
  • A track record of shipping production systems: APIs, containers, CI/CD and cloud (Azure or Kubernetes preferred).
  • The ability to explain evaluation concepts and results clearly, and to bring teams with you.
Desirable
  • Measuring developer productivity or the impact of engineering tooling.
  • Benchmarking coding agents on real‑world software tasks.
  • Red‑team­ing and AI safety testing.
  • Front office, market data or research experience at a hedge fund or bank.

BlueCrest is committed to providing an inclusive environment for its workforce.

As an employer, we provide equal opportunities to all people regardless of their gender, marital or civil partnership status, race, religion or ethnicity, disability, age, sexual orientation or nationality.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

AI Engineer: Evals
AI Engineer: Evals

Teli Labs • Greater London

Presencial
GBP 90.000 - 130.000
Research Engineer (Evals)
Research Engineer (Evals)

Moonfire • Greater London

Híbrido
GBP 90.000 - 130.000
Equity
Hybrid work model
Office in London/Paris
+3
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Stealth - AI x Bio • Greater London

Presencial
GBP 90.000 - 150.000
Senior AI Engineer
Senior AI Engineer

Betfred Group Holdings Limited • Manchester

Presencial
GBP 70.000 - 120.000
AI Engineer: Evals
AI Engineer: Evals

PhysicsX • Greater London

Presencial
GBP 90.000 - 120.000
Research Engineer, Benchmarking - Member of Technical Staff
Research Engineer, Benchmarking - Member of Technical Staff

Callosum Technologies Ltd. • Greater London

Presencial
GBP 75.000 - 120.000
Visa sponsorship
Relocation benefits
Research Engineer, Benchmarking - Member of Technical Staff
Research Engineer, Benchmarking - Member of Technical Staff

AI Startups UK • Greater London

Presencial
GBP 120.000 - 180.000
Competitive salary
Equity & ownership
Private healthcare
+2
Software Engineer - AI Evaluation
Software Engineer - AI Evaluation

Board Intelligence • Greater London

Híbrido
GBP 70.000 - 100.000
Pension scheme
Personal performance bonus
26 days holiday each calendar year
+4
AI System Engineer
AI System Engineer

Cognizant • Greater London

Presencial
GBP 90.000 - 140.000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Ai • Greater London

Presencial
GBP 85.000 - 130.000
30 days annual leave
Pension plan with 10% employer contrib
Bupa private health cover
+1