Lead Quality Assurance Engineer

RVS; Colombia; Remote; COP 120,000,000 - 180,000,000; Full time

Job summary

RV Holdings S.A.S. seeks an experienced Lead QA Engineer to own AI/LLM testing, validate agentic workflows, and drive QA maturity in a remote team. You will design test plans, execute manual and automated tests, and partner with development and product to ensure reliable releases.

Excellent knowledge of QA tooling (Jira, TestRail) and data validation (SQL, MongoDB) is required, plus hands-on automation with Playwright, Cypress, or pytest. Strong communication with US stakeholders is essential.

Qualifications

• Bachelor’s degree in CS/IT or equivalent experience.
• 3+ years of professional QA experience.
• Hands-on experience testing AI agents and LLM-based apps.
• Familiarity with QA tooling, CI/CD, and data testing challenges.

Responsibilities

• Design and execute test cases for AI agents and agentic workflows.
• Plan, execute, and automate manual and regression tests.
• Lead QA practices and improve test coverage across development.
• Identify, document, and track defects with clear steps.
• Develop and maintain automated tests with Playwright, Selenium, Cypress, or pytest.
• Perform API testing for backend services and integrations.
• Validate data via SQL/MongoDB queries and data investigations.
• Collaborate with devs and product stakeholders to meet acceptance criteria.
• Support biweekly releases, including evening windows for validation.

Skills

QA Testing

AI Testing

English Proficiency

Remote Work

Test Planning

Education

BSc in CS/IT

Tools

Jira

TestRail

SQL

MongoDB

Databricks

Git

CI/CD

Playwright

Selenium WebDriver

Cypress

pytest

Job description

RVStaffing is seeking an experienced Lead QA Engineer with hands-on experience testing AI agents, LLM-based applications, and agentic workflows. This role will focus on ensuring the quality, reliability, and consistency of AI-driven systems through structured testing, defect investigation, and continuous improvement of QA processes.

The ideal candidate combines strong software testing fundamentals with practical experience validating AI-powered applications, identifying unexpected behavior, and collaborating with engineering and product teams to deliver reliable solutions.

Please note: This position requires flexible working hours to support scheduled releases every two weeks, beginning at approximately 9:00 PM Pacific Standard Time (PST). Availability during these release windows is required.

What You’ll Do

• AI & Agentic Testing: Design and execute test cases to validate AI agents, LLM-based applications, and agentic workflows, including expected behavior, edge cases, failure scenarios, and consistency of results.
• Test Planning & Execution: Develop and execute manual and automated test cases for new features, functional requirements, and regression testing.
• Quality Assurance Leadership: Identify gaps in test coverage, recommend process improvements, and strengthen QA practices across the development lifecycle.
• Defect Management: Identify, document, prioritize, and track defects with clear reproduction steps, expected versus actual results, and severity levels.
• Automation Support: Develop, maintain, or improve automated test scripts using frameworks such as Playwright, Selenium WebDriver, Cypress, or pytest.
• API Testing: Perform manual and automated API testing to validate backend services, integrations, and application behavior.
• Data Validation: Use database queries and data investigation techniques to validate application behavior and troubleshoot defects.
• Cross-Functional Collaboration: Work closely with developers and product stakeholders to clarify requirements, troubleshoot issues, and ensure acceptance criteria are met.
• Release Support: Participate in biweekly release cycles, including evening release windows, to validate functionality and support release quality.

What You’ll Bring to the Team

• Bachelor’s degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.
• Advanced / C1 English proficiency (fluent verbal and written communication required for daily collaboration with US stakeholders).
• 3+ years of professional experience in software quality assurance.
• Demonstrated hands-on experience testing AI agents, LLM-based applications, or agentic workflows is required.
• Understanding of AI-specific testing challenges, including non-deterministic outputs, response consistency, edge cases, and failure handling.
• Strong understanding of manual testing methodologies, test case design, defect lifecycle management, and regression testing.
• Experience identifying QA process improvements and gaps in test coverage.
• Proficiency with bug-tracking and test-management tools such as Jira and TestRail.
• Basic database skills for validating data and investigating defects, including SQL, MongoDB, and Databricks.
• Familiarity with Git and automated testing within CI/CD pipelines.
• Experience with at least one test automation framework, such as Playwright, Selenium WebDriver, Cypress, or pytest.
• Excellent written and verbal communication skills, with the ability to explain technical issues to both technical and non-technical stakeholders.
• Ability to work independently in a remote environment with minimal supervision.
• Availability for biweekly releases beginning around 9:00 PM PST and flexibility to work during these release windows are required.

What We’re Looking For

A proactive QA professional who combines traditional software testing expertise with practical AI testing experience. The successful candidate will take ownership of product quality, investigate unpredictable AI behavior, collaborate effectively across teams, and help ensure reliable releases in a fast-moving engineering environment.

At RV Holdings S.A.S., we promote an inclusive, diverse, and respectful work environment. We value talent regardless of race, ethnicity, gender, sexual orientation, gender identity, age, beliefs, nationality, or disability status. All selection processes are conducted based on the competencies, skills, and experience of each individual.