Senior AI Engineer

Madiff

Warszawa

Hybrid

PLN 120,000 - 180,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Hybrid work
Healthcare
Competitive salary
Long-term B2B contract

Job summary

Madiff is seeking a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform for an AI-powered shopping assistant. You will design and implement LLM evaluation pipelines, judge systems, golden datasets and automated tests, while analysing real conversations to improve safety and quality.

You will own the platform, collaborate with product and engineering teams, and enhance prompts, guardrails and monitoring to ensure reliable, measurable LLM performance in a

Qualifications

  • Minimum 5 years of software engineering experience.
  • Minimum 2 years working with LLM systems in production.
  • Strong Python development experience.
  • Hands-on experience evaluating LLM applications.
  • LLM-as-a-judge and evaluation frameworks.
  • Prompt engineering and context engineering.
  • Experience with AI agents, tool calling and multi-turn conversations.
  • LLM observability tools such as Langfuse, Braintrust, Weave or similar.
  • SQL and experience working with large datasets.
  • CI/CD, testing, Docker or other container technologies.
  • Cloud infrastructure experience.
  • Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot.
  • Good English and Polish.

Responsibilities

  • Develop and maintain LLM evaluation pipelines.
  • Build LLM-as-a-judge solutions to assess AI conversations.
  • Create evaluation datasets and golden test sets.
  • Analyse real customer conversations and identify recurring problems.
  • Measure AI response quality and compare model performance.
  • Build automated regression tests for LLM applications.
  • Develop monitoring and observability for AI systems.
  • Identify hallucinations, incorrect responses and unusual user behaviour.
  • Work on AI guardrails, adversarial testing and abuse detection.
  • Improve prompts, context management and agent workflows.
  • Work closely with product managers and engineering teams.
  • Take technical ownership of the evaluation platform.

Skills

Python
LLM systems
Prompt engineering
Context engineering
English & Polish

Tools

Langfuse
Braintrust
Weave
GitHub Copilot
Cursor

Job description

We are looking for a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform supporting a production AI-powered shopping assistant. The role focuses on building LLM evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave, while analysing real user conversations to identify hallucinations, quality issues, and potential risks. The engineer will take technical ownership of the evaluation platform, collaborate with product and engineering teams, and improve AI agents, prompt and context engineering, guardrails, and monitoring to ensure reliable, measurable, and continuously improving LLM performance.

Responsibilities
  • Develop and maintain LLM evaluation pipelines
  • Build LLM-as-a-judge solutions to assess AI conversations
  • Create evaluation datasets and golden test sets
  • Analyse real customer conversations and identify recurring problems
  • Measure AI response quality and compare model performance
  • Build automated regression tests for LLM applications
  • Develop monitoring and observability for AI systems
  • Identify hallucinations, incorrect responses and unusual user behaviour
  • Work on AI guardrails, adversarial testing and abuse detection
  • Improve prompts, context management and agent workflows
  • Work closely with product managers and engineering teams
  • Take technical ownership of the evaluation platform
Requirements
Must have
  • Minimum 5 years of software engineering experience
  • Minimum 2 years working with LLM systems in production
  • Strong Python development
  • Hands-on experience evaluating LLM applications
  • LLM-as-a-judge and evaluation frameworks
  • Prompt engineering and context engineering
  • Experience with AI agents, tool calling and multi-turn conversations
  • LLM observability tools such as Langfuse, Braintrust, Weave or similar
  • SQL and experience working with large datasets
  • CI/CD, testing, Docker or other container technologies
  • Cloud infrastructure experience
  • Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
  • Good English and Polish
Nice to have
  • Human evaluation and calibration of LLM results
  • Golden datasets and regression testing for generative AI
  • Agent orchestration and automated workflows
  • MCP and agent-to-agent integrations
  • AI red-teaming, jailbreak testing and guardrails
  • Experience improving LLM applications using production feedback
  • E-commerce, search or recommendation systems
  • Experience as a Technical Lead or Staff Engineer
Benefits
  • Flexible hybrid mode (1-2 days per week in the office in Warsaw/Kraków)
  • Solid, competitive salary
  • Work in a multinational environment on international projects
  • Comprehensive healthcare
  • Long-term B2B contract with a stable project pipeline
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Engineer / 1
AI Engineer / 1

Inetum • Województwo mazowieckie

Hybrid
PLN 180,000 - 240,000
Cafeteria system
Referral bonuses up to PLN6,000
Revenue sharing opportunities
Lead AI Engineer
Lead AI Engineer

EPAM Systems Inc • Poland

Hybrid
PLN 180,000 - 300,000
Hybrid work in Poland
Work abroad up to 60 days annually
Employee Stock Purchase Plan
Software Engineer (AI)
Software Engineer (AI)

SQUAD • Wrocław

On-site
PLN 180,000 - 300,000
Private medical insurance
Paid vacation and holidays
English language classes
+1
AI Engineer – LLM / Agentic Systems (f/m/x)
AI Engineer – LLM / Agentic Systems (f/m/x)

Sii Poland • Szczecin

On-site
PLN 180,000 - 300,000
Centre of internal trainings
Profit sharing
Medical care
Senior AI Software Engineer Python, LLM, AI, Docker, Kubernetes Gdynia
Senior AI Software Engineer Python, LLM, AI, Docker, Kubernetes Gdynia

DCG (Poland) • Gdynia

On-site
PLN 180,000 - 280,000
Private medical care
Co-financing for the sports card
Dedicated consultant
AI Engineer / 1
AI Engineer / 1

Inetum • Warszawa

Hybrid
PLN 180,000 - 260,000
Cafeteria benefits
Referral bonuses
Revenue sharing opportunities
+5
Senior AI Software Engineer Python, LLM, AI, Docker, Kubernetes Gdynia
Senior AI Software Engineer Python, LLM, AI, Docker, Kubernetes Gdynia

Diverse Consulting Group • Gdynia

On-site
PLN 180,000 - 280,000
Private medical care
Co-financing for sports card
Dedicated consultant support
AI Engineer / 1
AI Engineer / 1

Inetum • Województwo mazowieckie

Hybrid
PLN 180,000 - 240,000
Cafeteria benefits
Referral bonuses up to PLN 6,000
Revenue sharing opportunities
+1
Senior AI Platform Engineer
Senior AI Platform Engineer

EPAM Systems Inc • Poland

Hybrid
PLN 180,000 - 260,000
Hybrid by design
Remote work within Poland
Relocation opportunities
+7
AI Software Engineer
AI Software Engineer

Dotlinkers IT recruitment • Polska

Hybrid
PLN 262,000 - 314,000
Open PTO policy for eligible employees
Quarterly Days of Disconnect
Parental leave and pet-related leave
+5
Top locations