Senior Data Scientist — AI/LLM Evaluation & Metrics

Square One Resources; Poland; Remote; PLN 180,000 - 240,000; Full time

Job summary

Square One Poland is seeking a Senior Data Scientist to join an international project focused on AI, Large Language Models (LLMs), and AI evaluation. You will design evaluation metrics, build ground-truth datasets, analyze model performance, and advance automated and human-feedback-based evaluation approaches.

The role emphasizes evaluating AI agents and agentic workflows, improving LLM-based judges, and translating experimental results into actionable recommendations.

Qualifications

• Proven experience as a Senior Data Scientist or in a similar role.
• Strong data science, statistical analysis, and experimentation skills.
• Hands-on experience with AI/LLM evaluation.
• Strong understanding of LLMs and agentic workflows.
• Experience designing, validating, and optimizing evaluation metrics for AI/ML models.
• Strong analytical skills and experience investigating model errors, including false positives and false negatives.
• Experience with ground-truth datasets and defining quality criteria.
• Ability to independently analyze experimental results and translate findings into actionable recommendations.
• Strong problem-solving and analytical mindset.

Responsibilities

• Design and validate evaluation metrics and ground-truth datasets.
• Analyze evaluation quality, including accuracy, false positives, and false negatives.
• Optimize metric normalization and scoring methodologies.
• Develop and improve LLM-based judges for automated evaluation.
• Design and implement human-feedback-based evaluation approaches.
• Explore and develop new evaluation metrics, including: (list items to follow)

Skills

Statistical analysis

AI

LLM

Agentic workflow

3D

Geometry

LLM-as-a-Judge

LLM-based evaluation

Evaluation metrics

Ground-truth datasets

Human-feedback eval

Job description

Square One Poland is seeking a Senior Data Scientist to join an international project focused on AI, Large Language Models (LLMs), and AI evaluation. You will design evaluation metrics, build ground-truth datasets, analyze model performance, and advance automated and human-feedback-based evaluation approaches.

The role emphasizes evaluating AI agents and agentic workflows, improving LLM-based judges, and translating experimental results into actionable recommendations.