Senior QA Engineer for AI/LLM Apps — Remote

Ascendix Technologies; United States; Remote; USD 110,000 - 160,000; Full time

Benefits offered by this job

20 paid days-off

5 paid sick days

Remote cooperation

Career development and mentoring

Internal knowledge-sharing

Team events and open management

Job summary

Ascendix Technologies is seeking an experienced QA Engineer to own QA for Salesforce-related products, starting with the Ascendix AI Agent. You will combine manual testing, test automation, and LLM evaluation to ensure reliable AI-driven functionality and traditional software quality, collaborating with a small distributed team.

The role emphasizes ownership, proactive process improvement, and strong API testing for robust product releases.

Qualifications

• 5+ years of experience in Software Quality Assurance.
• Hands-on testing of AI-powered apps with non-deterministic outcomes (LLMs, AI agents, chatbots).
• Practical experience with LLM evaluation techniques, including scoring rubrics and regression tracking.
• Experience in test automation and maintaining automated test suites.
• Strong API testing skills using Postman or similar tools.
• Analytical and critical thinking to evaluate AI outputs and underlying causes.
• Ownership mindset and proactive QA process improvements.
• Good communication and collaboration in a distributed team.
• Upper-Intermediate English (B2+) or higher.

Responsibilities

• Take ownership of product quality across Salesforce-related solutions, focusing on Ascendix AI Agent.
• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic features.
• Evaluate AI-generated responses for accuracy, hallucinations, and correct action execution.
• Design and maintain LLM evaluation datasets, scoring criteria, and regression tests.
• Develop and maintain automated tests using Playwright and AI-assisted tools.
• Collaborate with developers and Business Analyst to define clear acceptance criteria.
• Improve QA processes and document AI agent skills and workflows in Markdown.
• Investigate and reproduce issues, delivering actionable bug reports.
• Identify quality risks and suggest testing improvements.

Skills

QA Experience

AI testing

LLM evaluation

Test automation

Communication

Ownership mindset

Tools

Playwright

Postman

Langfuse

Promptfoo

DeepEval

Job description

Ascendix Technologies is seeking an experienced QA Engineer to own QA for Salesforce-related products, starting with the Ascendix AI Agent. You will combine manual testing, test automation, and LLM evaluation to ensure reliable AI-driven functionality and traditional software quality, collaborating with a small distributed team.

The role emphasizes ownership, proactive process improvement, and strong API testing for robust product releases.