Senior AI Engineer

Madiff; Województwo mazowieckie; Híbrido; PLN 320.000 - 520.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Comprehensive healthcare

Flexible hybrid mode (1–2 days in the

Long-term B2B contract with a stableプロ

Descripción de la vacante

Madiff is seeking a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform for a production AI-powered shopping assistant. You will build evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave.

You will analyse real user conversations to identify hallucinations, quality issues, and risks, taking technical ownership of the platform while

Formación

• Minimum 5 years of software engineering experience.
• Minimum 2 years working with LLM systems in production.
• Experience with LLM-as-a-judge and evaluation frameworks.
• Prompt engineering and context engineering.
• Experience with AI agents, tool calling and multi-turn conversations.
• LLM observability tools such as Langfuse, Braintrust, Weave or similar.
• SQL and experience working with large datasets.
• CI/CD, testing, Docker or other container technologies.
• Cloud infrastructure experience.
• Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot.
• Good English and Polish.

Responsabilidades

• Develop and maintain LLM evaluation pipelines.
• Build LLM-as-a-judge solutions to assess AI conversations.
• Create evaluation datasets and golden test sets.
• Analyse real customer conversations and identify recurring problems.
• Measure AI response quality and compare model performance.
• Build automated regression tests for LLM applications.
• Develop monitoring and observability for AI systems.
• Identify hallucinations, incorrect responses and unusual user behaviour.
• Work on AI guardrails, adversarial testing and abuse detection.
• Improve prompts, context management and agent workflows.
• Work closely with product managers and engineering teams.
• Take technical ownership of the evaluation platform.

Conocimientos

Software engineering

LLM systems in production

Prompt engineering

Context engineering

AI agents and tool calling

Observability in AI

SQL

CI/CD and Docker

English and Polish

Herramientas

Langfuse

Braintrust

Weave

Cursor

Claude Code

GitHub Copilot

Descripción del empleo

We are looking for a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform supporting a production AI-powered shopping assistant. The role focuses on building LLM evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave, while analysing real user conversations to identify hallucinations, quality issues, and potential risks. The engineer will take technical ownership of the evaluation platform, collaborate with product and engineering teams, and improve AI agents, prompt and context engineering, guardrails, and monitoring to ensure reliable, measurable, and continuously improving LLM performance.

Responsibilities

• Develop and maintain LLM evaluation pipelines
• Build LLM-as-a-judge solutions to assess AI conversations
• Create evaluation datasets and golden test sets
• Analyse real customer conversations and identify recurring problems
• Measure AI response quality and compare model performance
• Build automated regression tests for LLM applications
• Develop monitoring and observability for AI systems
• Identify hallucinations, incorrect responses and unusual user behaviour
• Work on AI guardrails, adversarial testing and abuse detection
• Improve prompts, context management and agent workflows
• Work closely with product managers and engineering teams
• Take technical ownership of the evaluation platform

Requirements

Must have

• Minimum 5 years of software engineering experience
• Minimum 2 years working with LLM systems in production
• LLM-as-a-judge and evaluation frameworks
• Prompt engineering and context engineering
• Experience with AI agents, tool calling and multi-turn conversations
• LLM observability tools such as Langfuse, Braintrust, Weave or similar
• SQL and experience working with large datasets
• CI/CD, testing, Docker or other container technologies
• Cloud infrastructure experience
• Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
• Good English and Polish

Nice to have

• Human evaluation and calibration of LLM results
• Golden datasets and regression testing for generative AI
• Agent orchestration and automated workflows
• MCP and agent-to-agent integrations
• AI red-teaming, jailbreak testing and guardrails
• Experience improving LLM applications using production feedback
• Experience as a Technical Lead or Staff Engineer
• Flexible hybrid mode ( 1–2 days per week in the office in Warsaw/Kraków)
• Work in a multinational environment on international projects
• Comprehensive healthcare
• Long-term B2B contract with a stable project pipeline