Gen-AI QA Data Scientist | Python, LLM Evaluation, German

Jobtailor; Kraków; On-site; PLN 150,000 - 220,000; Full time

Job summary

Jobtailor in Kraków, Poland, seeks an engineer to develop automated evaluation pipelines for agentic AI. You will craft datasets, benchmarks, and synthetic tests, while defining measurable client acceptance criteria and reporting results in German.

The role requires strong Python, stats background, and hands-on CI/CD experience with modern tools. German proficiency is essential for client communications.

Qualifications

• Excellent Python skills; production-ready code using pytest, asyncio, pandas, and pydantic.
• Hands-on experience with agentic AI tools and LLM SDKs.
• Strong statistical background including hypothesis testing and sampling design.
• CI/CD experience with GitHub Actions or GitLab CI; basic IT security understanding.

Responsibilities

• Develop automated evaluation pipelines and test harnesses in Python.
• Build golden datasets and task-based benchmarks.
• Generate synthetic test cases and define acceptance criteria with clients.
• Evaluate results and trajectories of agents’ problem-solving processes.
• Assess tool selection, parameters, and solution paths for efficiency.
• Design LLM-as-a-Judge approaches and calibrate against human evals.
• Ensure statistical reliability of evaluation results.
• Operationalize quality metrics (Task Success Rate, Tool-Call Accuracy, Groundedness).
• Set up observability with Langfuse/LangSmith/OpenTelemetry.
• Integrate eval suites into CI/CD; conduct regression and red-team tests.

Skills

Python programming

Pytest

Asyncio

Pydantic

Statistical analysis

Hypothesis testing

Sampling design

Significance testing

CI/CD

Agent frameworks

Tools

Langfuse

LangSmith

OpenTelemetry

GitHub Actions

GitLab CI

KNIME

Power BI

Claude Code

Anthropic SDK

OpenAI SDK

Job description

Jobtailor in Kraków, Poland, seeks an engineer to develop automated evaluation pipelines for agentic AI. You will craft datasets, benchmarks, and synthetic tests, while defining measurable client acceptance criteria and reporting results in German.

The role requires strong Python, stats background, and hands-on CI/CD experience with modern tools. German proficiency is essential for client communications.