QA Engineer (AI products)

Ascendix Technologies; United States; Remote; USD 110,000 - 160,000; Full time

Benefits offered by this job

20 paid days-off

5 paid sick days

Remote cooperation

Career development and mentoring

Internal knowledge-sharing

Team events and open management

Job summary

Ascendix Technologies is seeking an experienced QA Engineer to own QA for Salesforce-related products, starting with the Ascendix AI Agent. You will combine manual testing, test automation, and LLM evaluation to ensure reliable AI-driven functionality and traditional software quality, collaborating with a small distributed team.

The role emphasizes ownership, proactive process improvement, and strong API testing for robust product releases.

Qualifications

• 5+ years of experience in Software Quality Assurance.
• Hands-on testing of AI-powered apps with non-deterministic outcomes (LLMs, AI agents, chatbots).
• Practical experience with LLM evaluation techniques, including scoring rubrics and regression tracking.
• Experience in test automation and maintaining automated test suites.
• Strong API testing skills using Postman or similar tools.
• Analytical and critical thinking to evaluate AI outputs and underlying causes.
• Ownership mindset and proactive QA process improvements.
• Good communication and collaboration in a distributed team.
• Upper-Intermediate English (B2+) or higher.

Responsibilities

• Take ownership of product quality across Salesforce-related solutions, focusing on Ascendix AI Agent.
• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic features.
• Evaluate AI-generated responses for accuracy, hallucinations, and correct action execution.
• Design and maintain LLM evaluation datasets, scoring criteria, and regression tests.
• Develop and maintain automated tests using Playwright and AI-assisted tools.
• Collaborate with developers and Business Analyst to define clear acceptance criteria.
• Improve QA processes and document AI agent skills and workflows in Markdown.
• Investigate and reproduce issues, delivering actionable bug reports.
• Identify quality risks and suggest testing improvements.

Skills

QA Experience

AI testing

LLM evaluation

Test automation

Communication

Ownership mindset

Tools

Playwright

Postman

Langfuse

Promptfoo

DeepEval

Job description

Ascendix Technologies is an AI and Real Estate Technology partner headquartered in Dallas, Texas, with offices in Poland, Portugal, and Ukraine. We specialize in AI solutions, Salesforce, and custom software development, helping businesses streamline their operations through innovative technology.

We're looking for an experienced QA Engineer to join our Product Department and take ownership of quality assurance for our Salesforce-related products, starting with the Ascendix AI Agent.

In this role, you'll work with a small, collaborative team to ensure the reliability of both traditional software functionality and AI-driven features. You'll combine manual testing, test automation, and LLM evaluation techniques to validate AI-generated responses, identify potential issues, and continuously improve product quality.

If you enjoy exploring AI technologies, thinking critically about complex scenarios, and taking initiative in shaping QA processes, we'd love to hear from you.

WhatWe're Looking For

Requirements:

• 5+ years of experience in Software Quality Assurance;

• Hands-on experience testing AI-powered applications with non-deterministic outcomes, such as LLM-based products, AI agents, or chatbots;

• Practical experience with LLM evaluation techniques, including test datasets, scoring rubrics, LLM-as-judge approaches, and regression tracking;

• Experience in test automation and maintaining automated test suites;

• Solid API testing skills using Postman or similar tools;

• Strong analytical and critical thinking skills, with the ability to evaluate not only whether an AI-generated response is incorrect, but also why;

• Ownership mindset, proactive approach to problem-solving, and willingness to improve QA processes;

• Good communication and collaboration skills, with the ability to work effectively in a small, distributed team;

• Upper-Intermediate English (B2+) or higher, both written and spoken.

Niceto Have:

• Experience with Playwright (TypeScript or JavaScript);

• Salesforce testing experience, including Lightning, managed packages, and permission sets;

• Familiarity with LLM evaluation tools such as Promptfoo, DeepEval, Ragas, or Langfuse;

• Experience using AI-assisted development tools such as Claude Code, Cursor, or similar;

• Basic prompt engineering skills;

• Experience working with CRM systems or real estate data.

WhatYou’ll Do

Responsibilities:

• Take ownership of product quality across our Salesforce-related solutions, initially focusing on the Ascendix AI Agent;

• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic functionality;

• Evaluate AI-generated responses for accuracy, hallucinations, and correct execution of actions;

• Design, maintain, and improve LLM evaluation datasets, scoring criteria, and regression tests;

• Develop and maintain automated tests using Playwright and modern AI-assisted tools;

• Collaborate with developers and the Business Analyst in a spec-driven development process, reviewing specifications and defining clear acceptance criteria;

• Establish and continuously improve QA processes, including maintaining AI agent skills, rules, and workflow documentation in Markdown files;

• Investigate and reproduce issues reported by clients, providing clear and actionable bug reports;

• Proactively identify quality risks and suggest improvements to product reliability and testing practices.

AscendixOffers You:

Healthcare:

• 20 paid days-off;

• 5 paid sick days.

Career and Professional Growth:

• B2B and remote cooperation;

• Career development and professional growth;

• Technical performance reviews and 360 Evaluation;

• Collaboration with a personal Direct Manager, Technical Leads, and mentors.

Learning and Development:

• Opportunity for external event compensation;

• Internal knowledge-sharing opportunities.

Corporate Spirit:

• Online and offline events and team buildings;

• Friendly and supportive atmosphere;

• Open management.

Originally posted on Himalayas