Ssr QA Engineer

Rootstrap; Argentina; Remote; ARS 106,019,000 - 166,601,000; Full time

Benefits offered by this job

Certifications

Mentorship Program

Learning Bonus

Workation program

Gym benefits

Counseling services

Lunch reimbursements

Job summary

Rootstrap is seeking a Semi Senior QA Engineer to test AI-powered applications, including LLM-based features and RAG systems. You will design test plans, automate with Playwright, and build evaluation frameworks to measure quality and regressions.

The role emphasizes testing non-deterministic AI outputs, collaborating with engineers, and staying current with AI evaluation methods while enabling continuous QA improvements.

Qualifications

• 3+ years of QA engineering experience.
• Hands-on mobile testing.
• Experience with Playwright automation.
• Experience testing AI-powered apps and LLM-based systems.
• Advanced English communication skills.

Responsibilities

• Create detailed, comprehensive test plans and cases.
• Review specs with development teams to give feedback.
• Estimate, prioritize, plan, and coordinate testing activities.
• Identify, document, and track bugs.
• Design and maintain Playwright automation scripts.
• Perform regression testing when bugs are fixed.
• Track QA metrics and reports.
• Maintain eval suites to validate AI features and detect regressions.
• Test non-deterministic AI systems including LLMs and AI agents.
• Analyze AI outputs to identify failure patterns and drive improvements.
• Stay up-to-date with QA tools and AI evaluation techniques.
• Investigate defect causes with the dev team and contribute to preventive solutions.
• Collaborate on internal initiatives.

Skills

QA testing

AI testing

Eval engineering

Automation testing

Automation scripting

Education

BSc in CS/SE

Tools

Playwright

LangSmith

OpenEvals

Arize Phoenix

CI/CD pipelines

Job description

Driven by our enthusiasm for meeting new talents and fostering connections within the community and as part of our profile mapping program, we are now exploring diverse roles, including Semi Senior QA Engineer to expand the QA area. It’s important to note that, in this instance, the role does not involve immediate hiring but rather serves as a part of a pipeline for future recruitments.

We’re looking for a QA Engineer with experience testing AI-powered applications. The ideal candidate has hands‑on experience evaluating non‑deterministic AI systems (LLMs, AI agents, and RAG applications) and building or maintaining evaluation frameworks (eval engineering) to measure quality, detect regressions, and improve AI system performance.

What are the main responsibilities of this role?

• Create detailed, comprehensive, and well‑structured test plans and test cases.

• Review client needs and technical specifications with the development team to provide timely and meaningful feedback.

• Estimate, prioritize, plan, and coordinate testing activities.

• Identify, record, document thoroughly, and track bugs.

• Design, develop, and execute automation scripts (including Playwright‑based automation).

• Perform thorough regression testing when bugs are resolved.

• Track quality assurance metrics and reports.

• Design and maintain evaluation suites (evals) to validate AI‑powered features and detect regressions in model behavior.

• Test non‑deterministic AI systems, including LLM‑based features, AI agents, and RAG applications, using qualitative and quantitative evaluation methods.

• Analyze AI outputs, identify failure patterns (hallucinations, instruction‑following issues, grounding, etc.), and collaborate with engineers to improve system quality.

• Stay up to date with new testing tools, AI evaluation techniques, and quality assurance strategies while driving continuous improvement.

• Investigate the causes of software defects and collaborate with the development team to implement preventive solutions.

• Collaborate on internal initiatives.

Which skills and experience do we envision for this role?

• Background in Software Engineering, Computer Science, or a related field.

• 3+ years of experience as a QA Engineer.

• Advanced English skills.

• Hands‑on experience with mobile testing (required).

• Experience with test automation, preferably using Playwright.

• Experience using AI tools (e.g., Claude or similar) to optimize testing processes, including test case creation, bug reporting, edge case identification, and automation support.

• Experience testing AI‑powered applications or LLM‑based systems.

• Understanding of the challenges of testing non‑deterministic systems and validating AI‑generated outputs.

• Experience with evaluation engineering, including building or maintaining evals, benchmark datasets, or automated AI quality evaluations.

Nice to have:

• Knowledge of CI/CD pipelines and automated test integration.

• Experience testing AI agents, RAG applications, or conversational AI systems.

• Familiarity with AI evaluation frameworks or LLM observability tools such as LangSmith, Braintrust, Arize Phoenix, Promptfoo, OpenEvals, or similar.

Join our team and enjoy:

• Access to Certifications, and a Mentorship Program for your professional growth.

• Learning Bonus.

• Opportunities to organize cross‑functional initiatives and receive 360 Feedback to improve your skills continuously.

• The flexibility to work remotely or from our offices in Montevideo, Buenos Aires, and Medellín, with a flexible time schedule and Workation program.

• Gym benefits, psychological counseling, and weekly lunch reimbursements with special foods in the offices.

• An Onboarding kit and access to cutting‑edge technologies and tools to make your work easier.

• After offices, Prizes, and special occasions gifts to celebrate your achievements.

• People Care referent to support your well‑being and personal development.

We value your well‑being and happiness.We grow together!

How we work:

• We offer a flexible and diverse work environment that fosters multicultural talent

• We value autonomy and creativity, We want people to bring ideas to the table and to think as leaders.

• We put our focus on real human connections

• We have a commitment to excellence

At Rootstrap, we help companies launch, grow, and scale their digital products and services. With over 160 global team members, we’ve built solutions for industry leaders like MasterClass, Universal, and Google, as well as for innovative startups and inspired pioneers. Our team takes pride in making an impact that goes beyond the code, earning regular recognition as a Fastest Growing Company and Best B2B Service Provider.

We embrace an AI‑first mindset, continuously exploring how artificial intelligence can enhance the way we build products, solve problems, and deliver value. Across all stages of the development process, we encourage our teams to actively leverage AI to boost productivity and drive innovation.

With offices in Uruguay, Argentina, and Colombia, we offer the flexibility to work remotely. Discover more about life at Rootstrap at https://www.rootstrap.com/careers .

All of our candidate searches are covered under Uruguayan Law 19691 (the purpose of this law is to promote the inclusion of people with disabilities).

As part of our Security and Compliance process, all seniority levels and roles must follow the company’s information security policies and secure development practices, and promptly report any suspected security incidents or vulnerabilities.