QA Engineer (AI products)

Ascendix; Kraków; On-site; PLN 140,000 - 210,000; Full time

Benefits offered by this job

20 paid days-off

5 paid sick days

B2B and remote cooperation

Career development and professional成长

Technical performance reviews and 360°

External event compensation

Internal knowledge-sharing

Online and offline events and teamings

Open management

Job summary

Ascendix Technologies is seeking an experienced QA Engineer to own quality assurance for Salesforce-related products, starting with the Ascendix AI Agent. You will combine manual testing, test automation, and LLM evaluation techniques to validate AI-driven features and ensure reliability across both traditional software and AI-enabled capabilities.

The role requires 5+ years in QA, hands-on AI testing experience, and strong API testing skills.

Qualifications

• 5+ years of experience in Software Quality Assurance.
• Hands-on experience testing AI-powered applications with non-deterministic outcomes, such as LLM-based products, AI agents, or chatbots.
• Practical experience with LLM evaluation techniques, including test datasets, scoring rubrics, LLM-as-judge approaches, and regression tracking.
• Experience in test automation and maintaining automated test suites.
• Solid API testing skills using Postman or similar tools.
• Strong analytical and critical thinking skills, with the ability to evaluate not only whether an AI-generated response is incorrect, but also why.
• Ownership mindset, proactive approach to problem-solving, and willingness to improve QA processes.
• Good communication and collaboration skills, with the ability to work effectively in a small, distributed team.
• Upper-Intermediate English (B2+) or higher, both written and spoken.

Responsibilities

• Take ownership of product quality across our Salesforce-related solutions, initially focusing on the Ascendix AI Agent.
• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic functionality.
• Evaluate AI-generated responses for accuracy, hallucinations, and correct execution of actions.
• Design, maintain, and improve LLM evaluation datasets, scoring criteria, and regression tests.
• Develop and maintain automated tests using Playwright and modern AI-assisted tools.
• Collaborate with developers and the Business Analyst in a spec-driven development process, reviewing specifications and defining clear acceptance criteria.
• Establish and continuously improve QA processes, including maintaining AI agent skills, rules, and workflow documentation in Markdown files.
• Investigate and reproduce issues reported by clients, providing clear and actionable bug reports.
• Proactively identify quality risks and suggest improvements to product reliability and testing practices.

Skills

Software QA

AI-powered testing

LLM evaluation

Test automation

API testing

Analytical thinking

Ownership mindset

Communication skills

English (B2+)

Tools

Postman

Playwright (TS/JS)

Promptfoo

Langfuse

DeepEval

Claude Code

Job description

Ascendix Technologies is an AI and Real Estate Technology partner headquartered in Dallas, Texas, with offices in Poland, Portugal, and Ukraine. We specialize in AI solutions, Salesforce, and custom software development, helping businesses streamline their operations through innovative technology.

We're looking for an experienced QA Engineer to join our Product Department and take ownership of quality assurance for our Salesforce-related products, starting with the Ascendix AI Agent.

In this role, you'll work with a small, collaborative team to ensure the reliability of both traditional software functionality and AI-driven features. You'll combine manual testing, test automation, and LLM evaluation techniques to validate AI-generated responses, identify potential issues, and continuously improve product quality.

If you enjoy exploring AI technologies, thinking critically about complex scenarios, and taking initiative in shaping QA processes, we'd love to hear from you.

What We're Looking For

Requirements:

• 5+ years of experience in Software Quality Assurance;
• Hands-on experience testing AI-powered applications with non-deterministic outcomes, such as LLM-based products, AI agents, or chatbots;
• Practical experience with LLM evaluation techniques, including test datasets, scoring rubrics, LLM-as-judge approaches, and regression tracking;
• Experience in test automation and maintaining automated test suites;
• Solid API testing skills using Postman or similar tools;
• Strong analytical and critical thinking skills, with the ability to evaluate not only whether an AI-generated response is incorrect, but also why;
• Ownership mindset, proactive approach to problem-solving, and willingness to improve QA processes;
• Good communication and collaboration skills, with the ability to work effectively in a small, distributed team;
• Upper-Intermediate English (B2+) or higher, both written and spoken.

Nice to Have:

• Experience with Playwright (TypeScript or JavaScript);
• Salesforce testing experience, including Lightning, managed packages, and permission sets;
• Familiarity with LLM evaluation tools such as Promptfoo, DeepEval, Ragas, or Langfuse;
• Experience using AI-assisted development tools such as Claude Code, Cursor, or similar;
• Basic prompt engineering skills;
• Experience working with CRM systems or real estate data.

What You'll Do

Responsibilities:

• Take ownership of product quality across our Salesforce-related solutions, initially focusing on the Ascendix AI Agent;
• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic functionality;
• Evaluate AI-generated responses for accuracy, hallucinations, and correct execution of actions;
• Design, maintain, and improve LLM evaluation datasets, scoring criteria, and regression tests;
• Develop and maintain automated tests using Playwright and modern AI-assisted tools;
• Collaborate with developers and the Business Analyst in a spec-driven development process, reviewing specifications and defining clear acceptance criteria;
• Establish and continuously improve QA processes, including maintaining AI agent skills, rules, and workflow documentation in Markdown files;
• Investigate and reproduce issues reported by clients, providing clear and actionable bug reports;
• Proactively identify quality risks and suggest improvements to product reliability and testing practices.

Ascendix Offers You:

Healthcare:

• 20 paid days-off;
• 5 paid sick days.

Career and Professional Growth:

• B2B and remote cooperation;
• Career development and professional growth;
• Technical performance reviews and 360 Evaluation;
• Collaboration with a personal Direct Manager, Technical Leads, and mentors.

Learning and Development:

• Opportunity for external event compensation;
• Internal knowledge-sharing opportunities.

Corporate Spirit:

• Online and offline events and team buildings;
• Friendly and supportive atmosphere;
• Open management.