Technical Lead — AI Evaluation & Platform Architect

InPost; Kraków; Hybrid; PLN 380,000 - 680,000; Full time

Benefits offered by this job

Hybrid work (Kraków/Warsaw)

Flexible hours

Training opportunities

B2B contract

Job summary

InPost seeks a Technical Lead, Senior AI Engineer to own the Evaluations Platform for VonHalsky, InPost's conversational AI assistant. You will define technical direction, mentor engineers, and ensure production-grade evals before features reach users.

You will drive the LLM-judge framework, curate real and golden data, and maintain robust diagnostics while aligning with product goals and safety standards.

Qualifications

• 5+ years building and running production software with 2+ years on LLM-based systems used by real users.
• Strong engineering fundamentals with production ownership (CI/CD, testing, containers, observability, cloud).
• Experience evaluating generative systems: LLM-as-judge, calibration, regression suites, offline-online comparisons.
• AI engineering fundamentals: prompt/context design, multi-turn state, tool use, agent patterns, observability tooling.
• Comfort with data at scale: SQL, data lakes/warehouses, reproducible metrics.
• Fluency with AI-assisted development tools (e.g., Claude Code, Copilot).
• Ability to explain technical concepts to non-technical audiences.
• English B2 and Polish language proficiency.

Responsibilities

• Own the LLM-judge pipeline and ensure gate decisions are trusted for releases.
• Develop eval datasets and a robust golden set with wide Polish-language coverage.
• Perform root-cause analysis on real conversations and improve diagnostic tooling.
• Maintain the “sus” detector and guardrails alongside red-team work.
• Drive eval-driven development: define tests before implementing features.
• Serve as interface to product and business with measurable, actionable results.

Skills

LLM systems

Production software

Engineering fundamentals

AI engineering

Data at scale

AI tooling

Non-technical communication

English proficiency

Tools

Python

LangFuse

Braintrust

Weave

Job description

InPost seeks a Technical Lead, Senior AI Engineer to own the Evaluations Platform for VonHalsky, InPost's conversational AI assistant. You will define technical direction, mentor engineers, and ensure production-grade evals before features reach users.

You will drive the LLM-judge framework, curate real and golden data, and maintain robust diagnostics while aligning with product goals and safety standards.