GenAI Quality Assurance Data Scientist

Jobtailor; Kraków; On-site; PLN 120,000 - 200,000; Full time

Job summary

Jobtailor in Kraków, Poland, seeks a GenAI evaluation engineer to develop automated evaluation pipelines and Python test harnesses for reproducibly assessing agent behavior, build golden datasets, and generate synthetic test cases.

You will work with clients to define measurable acceptance criteria, evaluate problem-solving trajectories, and communicate findings in German for specialists and management, while integrating quality gates into CI/CD workflows.

Qualifications

• Excellent Python skills and production-ready code experience with pytest, asyncio, pandas, and pydantic.
• Native-level German proficiency (C2).
• Strong proficiency with agentic AI development tools such as Claude Code.
• Hands-on experience with LLM SDKs such as Anthropic and OpenAI.
• Experience with agent frameworks such as LangGraph.
• Solid foundation in statistics including hypothesis testing, sampling design, and significance testing.
• Hands-on CI/CD experience with GitHub Actions or GitLab CI.
• Basic understanding of IT security.
• Experience with KNIME or Power BI.

Responsibilities

• Develop automated evaluation pipelines and Python test harnesses for reproducibly assessing GenAI agent behavior.
• Build golden datasets and task-based benchmarks.
• Generate synthetic test cases.
• Work with clients to define measurable acceptance criteria.
• Evaluate agent problem-solving trajectories, including tool selection, parameter validity, and solution-path efficiency.
• Design and calibrate LLM-as-a-Judge approaches against human evaluations.
• Ensure statistical reliability of evaluation results.
• Operationalize quality metrics including Task Success Rate, Tool-Call Accuracy, Groundedness, and Robustness.
• Include confidence intervals and sound sampling designs in quality measurement.
• Set up observability and tracing solutions using tools such as Langfuse, LangSmith, or OpenTelemetry.
• Develop error taxonomies and monitor quality in production environments.
• Integrate evaluation suites as quality gates into CI/CD pipelines.
• Conduct regression testing when prompts, tools, or models change.
• Conduct red-team testing, including prompt injection and guardrail testing.
• Document testing procedures with EU AI Act regulatory requirements in mind.
• Prepare and communicate findings in German for clients’ specialist departments and management teams.

Skills

Python programming

German proficiency

Agentic AI tools

LLM SDKs

LangGraph

Statistics

CI/CD

IT security basics

Power BI

Tools

Claude Code

Anthropic SDK

OpenAI SDK

Langfuse

OpenTelemetry

GitHub Actions

GitLab CI

KNIME

Power BI

Job description

Jobtailor in Kraków, Poland, seeks a GenAI evaluation engineer to develop automated evaluation pipelines and Python test harnesses for reproducibly assessing agent behavior, build golden datasets, and generate synthetic test cases.

You will work with clients to define measurable acceptance criteria, evaluate problem-solving trajectories, and communicate findings in German for specialists and management, while integrating quality gates into CI/CD workflows.