Lead Quality Assurance Engineer

RVS; Colombia; Remote; COP 60,000,000 - 90,000,000; Full time

Job summary

RVStaffing is seeking an experienced Lead QA Engineer to test AI agents, LLM-based applications, and agentic workflows. You will lead QA activities, design test plans, and ensure reliability across releases in a fast-paced AI environment.

You will collaborate with engineering and product teams, perform API and data validation, and contribute to QA process improvements. This role demands evening release windows and strong communication in a remote setup.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field, or equivalent practical experience.
• 3+ years of professional experience in software quality assurance.
• Hands-on experience testing AI agents, LLM-based applications, or agentic workflows.
• Understanding of AI-specific testing challenges, including non-deterministic outputs, response consistency, edge cases, and failure handling.
• Strong understanding of manual testing methodologies, test case design, defect lifecycle management, and regression testing.
• Experience identifying QA process improvements and gaps in test coverage.
• Proficiency with bug-tracking and test-management tools such as Jira and TestRail.
• Basic database skills for validating data and investigating defects, including SQL, MongoDB, and Databricks.
• Familiarity with Git and automated testing within CI/CD pipelines.
• Experience with at least one test automation framework, such as Playwright, Selenium WebDriver, Cypress, or pytest.
• Excellent written and verbal communication skills, with the ability to explain technical issues to both technical and non-technical stakeholders.
• Ability to work independently in a remote environment with minimal supervision.
• Availability for biweekly releases beginning around 9:00 PM PST and flexibility to work during these release windows are required.

Responsibilities

• AI & Agentic Testing: Design and execute test cases to validate AI agents and agentic workflows, including edge cases and failure scenarios.
• Test Planning & Execution: Develop and execute manual and automated test cases for new features and regression testing.
• Quality Assurance Leadership: Identify gaps in test coverage and recommend process improvements across the lifecycle.
• Defect Management: Document, prioritize, and track defects with clear reproduction steps and expected vs actual results.
• Automation Support: Develop and maintain automated test scripts using Playwright, Selenium WebDriver, Cypress, or pytest.
• API Testing: Perform manual and automated API testing to validate backend services.
• Data Validation: Use database queries to validate application behavior and troubleshoot defects.
• Cross-Functional Collaboration: Work with developers and product stakeholders to clarify requirements and ensure acceptance criteria.
• Release Support: Participate in biweekly release cycles, including evening windows, to validate functionality.

Skills

QA testing

Test planning

Cross-functional collaboration

Manual testing

Regression testing

Communication skills

Independent work

Release readiness

AI testing

API testing

Education

Bachelor's degree in Computer Science or related field

Tools

Jira

TestRail

Playwright

Selenium WebDriver

Cypress

pytest

SQL

MongoDB

Databricks

Git

Job description

RVStaffing is seeking an experienced Lead QA Engineer with hands-on experience testing AI agents, LLM-based applications, and agentic workflows. This role will focus on ensuring the quality, reliability, and consistency of AI-driven systems through structured testing, defect investigation, and continuous improvement of QA processes.

The ideal candidate combines strong software testing fundamentals with practical experience validating AI-powered applications, identifying unexpected behavior, and collaborating with engineering and product teams to deliver reliable solutions.

Please note: This position requires flexible working hours to support scheduled releases every two weeks, beginning at approximately 9:00 PM Pacific Standard Time (PST). Availability during these release windows is required.

What You'll Do

• AI & Agentic Testing: Design and execute test cases to validate AI agents, LLM-based applications, and agentic workflows, including expected behavior, edge cases, failure scenarios, and consistency of results.
• Test Planning & Execution: Develop and execute manual and automated test cases for new features, functional requirements, and regression testing.
• Quality Assurance Leadership: Identify gaps in test coverage, recommend process improvements, and strengthen QA practices across the development lifecycle.
• Defect Management: Identify, document, prioritize, and track defects with clear reproduction steps, expected versus actual results, and severity levels.
• Automation Support: Develop, maintain, or improve automated test scripts using frameworks such as Playwright, Selenium WebDriver, Cypress, or pytest.
• API Testing: Perform manual and automated API testing to validate backend services, integrations, and application behavior.
• Data Validation: Use database queries and data investigation techniques to validate application behavior and troubleshoot defects.
• Cross-Functional Collaboration: Work closely with developers and product stakeholders to clarify requirements, troubleshoot issues, and ensure acceptance criteria are met.
• Release Support: Participate in biweekly release cycles, including evening release windows, to validate functionality and support release quality.

What You'll Bring to the Team

• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.
• 3+ years of professional experience in software quality assurance.
• Demonstrated hands-on experience testing AI agents, LLM-based applications, or agentic workflows is required.
• Understanding of AI-specific testing challenges, including non-deterministic outputs, response consistency, edge cases, and failure handling.
• Strong understanding of manual testing methodologies, test case design, defect lifecycle management, and regression testing.
• Experience identifying QA process improvements and gaps in test coverage.
• Proficiency with bug-tracking and test-management tools such as Jira and TestRail.
• Basic database skills for validating data and investigating defects, including SQL, MongoDB, and Databricks.
• Familiarity with Git and automated testing within CI/CD pipelines.
• Experience with at least one test automation framework, such as Playwright, Selenium WebDriver, Cypress, or pytest.
• Excellent written and verbal communication skills, with the ability to explain technical issues to both technical and non-technical stakeholders.
• Ability to work independently in a remote environment with minimal supervision.
• Availability for biweekly releases beginning around 9:00 PM PST and flexibility to work during these release windows are required.

What We're Looking For

A proactive QA professional who combines traditional software testing expertise with practical AI testing experience. The successful candidate will take ownership of product quality, investigate unpredictable AI behavior, collaborate effectively across teams, and help ensure reliable releases in a fast-moving engineering environment.

At RV Holdings S.A.S., we promote an inclusive, diverse, and respectful work environment. We value talent regardless of race, ethnicity, gender, sexual orientation, gender identity, age, beliefs, nationality, or disability status. All selection processes are conducted based on the competencies, skills, and experience of each individual.