Senior AI Engineer - LLM Evaluation & Observability

Madiff; Województwo mazowieckie; Hybrid; PLN 320,000 - 520,000; Full time

Benefits offered by this job

Comprehensive healthcare

Flexible hybrid mode (1–2 days in the

Long-term B2B contract with a stableプロ

Job summary

Madiff is seeking a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform for a production AI-powered shopping assistant. You will build evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave.

You will analyse real user conversations to identify hallucinations, quality issues, and risks, taking technical ownership of the platform while

Qualifications

• Minimum 5 years of software engineering experience.
• Minimum 2 years working with LLM systems in production.
• Experience with LLM-as-a-judge and evaluation frameworks.
• Prompt engineering and context engineering.
• Experience with AI agents, tool calling and multi-turn conversations.
• LLM observability tools such as Langfuse, Braintrust, Weave or similar.
• SQL and experience working with large datasets.
• CI/CD, testing, Docker or other container technologies.
• Cloud infrastructure experience.
• Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot.
• Good English and Polish.

Responsibilities

• Develop and maintain LLM evaluation pipelines.
• Build LLM-as-a-judge solutions to assess AI conversations.
• Create evaluation datasets and golden test sets.
• Analyse real customer conversations and identify recurring problems.
• Measure AI response quality and compare model performance.
• Build automated regression tests for LLM applications.
• Develop monitoring and observability for AI systems.
• Identify hallucinations, incorrect responses and unusual user behaviour.
• Work on AI guardrails, adversarial testing and abuse detection.
• Improve prompts, context management and agent workflows.
• Work closely with product managers and engineering teams.
• Take technical ownership of the evaluation platform.

Skills

Software engineering

LLM systems in production

Prompt engineering

Context engineering

AI agents and tool calling

Observability in AI

SQL

CI/CD and Docker

English and Polish

Tools

Langfuse

Braintrust

Weave

Cursor

Claude Code

GitHub Copilot

Job description

Madiff is seeking a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform for a production AI-powered shopping assistant. You will build evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave.

You will analyse real user conversations to identify hallucinations, quality issues, and risks, taking technical ownership of the platform while