Senior QA Engineer - AI Agents & LLM Testing

RVS; Colombia; À distance; COP 60 000 000 - 90 000 000; Plein temps

Résumé du poste

RVStaffing is seeking an experienced Lead QA Engineer to test AI agents, LLM-based applications, and agentic workflows. You will lead QA activities, design test plans, and ensure reliability across releases in a fast-paced AI environment.

You will collaborate with engineering and product teams, perform API and data validation, and contribute to QA process improvements. This role demands evening release windows and strong communication in a remote setup.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field, or equivalent practical experience.
• 3+ years of professional experience in software quality assurance.
• Hands-on experience testing AI agents, LLM-based applications, or agentic workflows.
• Understanding of AI-specific testing challenges, including non-deterministic outputs, response consistency, edge cases, and failure handling.
• Strong understanding of manual testing methodologies, test case design, defect lifecycle management, and regression testing.
• Experience identifying QA process improvements and gaps in test coverage.
• Proficiency with bug-tracking and test-management tools such as Jira and TestRail.
• Basic database skills for validating data and investigating defects, including SQL, MongoDB, and Databricks.
• Familiarity with Git and automated testing within CI/CD pipelines.
• Experience with at least one test automation framework, such as Playwright, Selenium WebDriver, Cypress, or pytest.
• Excellent written and verbal communication skills, with the ability to explain technical issues to both technical and non-technical stakeholders.
• Ability to work independently in a remote environment with minimal supervision.
• Availability for biweekly releases beginning around 9:00 PM PST and flexibility to work during these release windows are required.

Responsabilités

• AI & Agentic Testing: Design and execute test cases to validate AI agents and agentic workflows, including edge cases and failure scenarios.
• Test Planning & Execution: Develop and execute manual and automated test cases for new features and regression testing.
• Quality Assurance Leadership: Identify gaps in test coverage and recommend process improvements across the lifecycle.
• Defect Management: Document, prioritize, and track defects with clear reproduction steps and expected vs actual results.
• Automation Support: Develop and maintain automated test scripts using Playwright, Selenium WebDriver, Cypress, or pytest.
• API Testing: Perform manual and automated API testing to validate backend services.
• Data Validation: Use database queries to validate application behavior and troubleshoot defects.
• Cross-Functional Collaboration: Work with developers and product stakeholders to clarify requirements and ensure acceptance criteria.
• Release Support: Participate in biweekly release cycles, including evening windows, to validate functionality.

Connaissances

QA testing

Test planning

Cross-functional collaboration

Manual testing

Regression testing

Communication skills

Independent work

Release readiness

AI testing

API testing

Formation

Bachelor's degree in Computer Science or related field

Outils

Jira

TestRail

Playwright

Selenium WebDriver

Cypress

pytest

SQL

MongoDB

Databricks

Git

Description du poste

RVStaffing is seeking an experienced Lead QA Engineer to test AI agents, LLM-based applications, and agentic workflows. You will lead QA activities, design test plans, and ensure reliability across releases in a fast-paced AI environment.

You will collaborate with engineering and product teams, perform API and data validation, and contribute to QA process improvements. This role demands evening release windows and strong communication in a remote setup.