AI Tester / AI Evaluation Engineer

Hiberus; Wrocław; Hybrid; PLN 75,000 - 110,000; Full time

Job summary

Hiberus in Poland is seeking an AI Tester / AI Evaluation Engineer to assess the quality and stability of AI, ML and GenAI solutions across diverse environments.

You will design test scenarios, create reference data, and evaluate model outputs for accuracy, completeness, and security while ensuring usability and compliance with requirements. Collaboration with AI engineers and security teams will be essential throughout solution acceptance.

Qualifications

• Minimum 3 years of experience in software testing or data-driven solutions.
• Knowledge of test case design and defect reporting.
• Basic understanding of ML or LLM models and non-deterministic systems testing.
• Ability to design test suites for evaluating model output quality.
• SQL and API testing experience with Postman.
• Ability to assess accuracy, completeness, source consistency, security and usability.

Responsibilities

• Prepare test scenarios for AI/ML/GenAI solutions.
• Design question sets, edge cases, and reference data for model evaluation.
• Test the quality of model outputs and stability of AI system behaviour.
• Perform regression testing after changes to the model, prompts, source data or configuration.
• Report defects, anomalies and quality-related risks.
• Collaborate with AI Engineers, analysts, business and security teams during solution acceptance.
• Document test results and prepare recommendations regarding AI system quality.

Skills

Software testing

Test case design

API testing

SQL

ML/LLM basics

Defect reporting

Non-deterministic testing

Tools

Postman

API testing tools

Job description

For our Client, we are looking for an AI Tester / AI Evaluation Engineer who will be responsible for testing and evaluating the quality and stability of AI, ML and GenAI-based solutions.

We are looking for a professional with experience in software testing or data-driven systems, who understands the specifics of ML/LLM models and can evaluate their outputs in terms of quality, compliance with requirements, security and usability.

Responsibilities

• Preparing test scenarios for AI, ML and GenAI solutions,
• Designing question sets, edge cases and reference data for model evaluation,
• Testing the quality of model outputs and the stability of AI system behaviour,
• Performing regression testing after changes to the model, prompts, source data or configuration,
• Reporting defects, anomalies and quality-related risks,
• Collaborating with AI Engineers, analysts, business and security teams during solution acceptance,
• Documenting test results and preparing recommendations regarding the quality of AI systems.

Experience required

• Minimum 3 years of experience in software testing, analytical systems or data-driven solutions,
• Practical knowledge of creating test cases, test scenarios and defect reports,
• Basic understanding of ML or LLM models and the specifics of testing non-deterministic systems,
• Ability to design test suites for evaluating model output quality, classification and information extraction,
• Knowledge of SQL, Postman and API testing tools,
• Ability to evaluate model outputs based on accuracy, completeness, source consistency, security and usability.

Nice to have

• Experience testing RAG systems, chatbots, AI assistants or predictive models,
• Knowledge of automated LLM evaluation tools and prompt regression testing,
• Experience in GenAI security testing, including prompt injection and jailbreak testing,
• Knowledge of accessibility, UX or plain-language requirements,
• English proficiency sufficient for analysing technical documentation.

About us

Who are we?

We are a team of experienced professionals from the IT, gaming, BI, project management and business sectors. Our team consists of people who have established and developed various types of companies and have been successful in the implementation of large projects. Among us are people with over 25 years of experience in the IT industry, as well as HR specialists with extensive knowledge in the field of training, administration and management.

Work with us

Job openings

We have a wide selection of IT jobs in different sectors and locations. We focus not only on matching skills, but also support you at every stage of employment.

If you have a question – call or write to us. Our specialists will try to answer as soon as possible.