AI Evaluation Engineer – LLMs & High-Impact AI Apps

HSBC Holdings plc; Kraków; On-site; PLN 195,000 - 293,000; Full time

Benefits offered by this job

Multisport card

Private medical care

Life insurance

Home office set-up reimbursement (up =

Job summary

HSBC Holdings plc in Kraków seeks an experienced AI/ML evaluation engineer to build robust evaluation approaches for LLM and AI applications in Credit and Lending, ensuring quality, reliability and safety before production.

You will define evaluation strategies, develop datasets and test harnesses, measure metrics, and collaborate with engineers, product managers and business experts to align coverage with user needs. Fluency in English and strong Python/ML skills are essential.

Qualifications

• Proven deep Python expertise in ML/AI
• Experience with FastAPI, LangChain and data containerisation
• Strong understanding of microservices, observability and API design
• Hands-on with Azure or GCP, containers, Kubernetes, and CI/CD tools (Jenkins, Azure DevOps, GCP Cloud Build)
• Experience with Test-Driven and Behavior-Driven Development and automated testing
• Solid grasp of LLM architectures, fine-tuning on domain data
• Hands-on Retrieval-Augmented Generation, advanced prompt engineering and agentic architectures
• Excellent English communication for technical and stakeholder discussions

Responsibilities

• Define and implement evaluation strategies for LLM/AI in Credit and Lending
• Build and maintain evaluation datasets and test harnesses for prompts, models, RAG and agentic workflows
• Measure metrics like accuracy, relevance, grounding, latency and cost with automated and human methods
• Evaluate retrieval quality, source attribution and grounding across financial data and vector databases
• Test AI systems for robustness, hallucinations, bias and safety risks; drive improvements
• Automate evaluation and regression tests in CI/CD before release
• Analyze results, identify quality regressions and communicate trade-offs to engineering and business
• Collaborate with engineers, product managers and business experts to align coverage with user needs

Skills

Python

ML/AI ecosystems

FastAPI

LangChain

Data containerisation

Microservices observability

APIs and concurrency

Azure/GCP deployments

Kubernetes

CI/CD tools (Jenkins, Azure DevOps, GC

Test-Driven Development

Behavior-Driven Development

LLM architectures & fine-tuning

Retrieval-Augmented Generation

Prompt engineering & agentic systems

English communication

Tools

FastAPI

LangChain

Data containers

Kubernetes

CI/CD (Jenkins)

Azure DevOps

GCP Cloud Build

Job description

HSBC Holdings plc in Kraków seeks an experienced AI/ML evaluation engineer to build robust evaluation approaches for LLM and AI applications in Credit and Lending, ensuring quality, reliability and safety before production.

You will define evaluation strategies, develop datasets and test harnesses, measure metrics, and collaborate with engineers, product managers and business experts to align coverage with user needs. Fluency in English and strong Python/ML skills are essential.