Senior AI Platform Lead – LLM Eval & Observability

InPost; Kraków; Hybride; PLN 320 000 - 520 000; Plein temps

Avantages offerts par ce poste

Hybrid work from Warsaw or Kraków

Training opportunities

Flexible hours

B2B contract

Excellent working environment

Résumé du poste

InPost is recruiting a technical lead to shape the Evaluations Platform for its conversational AI shopping assistant Von Halsky. You will set technical direction, review code, mentor engineers, and partner with PM/EM to transform vague requests into measurable, high-impact outcomes.

You will own the LLM-judge pipeline, eval datasets, root-cause analysis, and guardrails, driving eval-driven development across the team. Hybrid work from Kraków or Warsaw awaits.

Qualifications

• 5+ years building and running production software, with 2+ years on LLM-based systems.
• Strong engineering fundamentals including testing, CI/CD, containers, observability, and working in cloud.
• Experience evaluating generative systems: LLM-as-judge, calibration, regression suites.
• AI engineering fundamentals: prompt/context engineering, tool use, multi-turn state, eval observability.
• Data at scale: SQL, data lakes/warehouses.
• Fluency with AI-assisted development tooling.
• English B2 and Polish.

Responsabilités

• The LLM-judge pipeline development and gatekeeping.
• Eval datasets and the golden set coverage across Polish and other intents.
• Root-cause analysis on real conversations and diagnostic stack.
• The 'sus' detector and guardrails alignment.
• Eval-driven development: write evals before features.
• Interface to product and business with measurable outcomes.

Connaissances

Production software

LLM-based systems

Testing & CI/CD

Observability & cloud

SQL & data lakes

AI tooling proficiency

English & Polish

Outils

LangFuse

Braintrust

Weave

Python

SQL

Copilot/AI tooling

Description du poste

InPost is recruiting a technical lead to shape the Evaluations Platform for its conversational AI shopping assistant Von Halsky. You will set technical direction, review code, mentor engineers, and partner with PM/EM to transform vague requests into measurable, high-impact outcomes.

You will own the LLM-judge pipeline, eval datasets, root-cause analysis, and guardrails, driving eval-driven development across the team. Hybrid work from Kraków or Warsaw awaits.