Semi-Senior AI QA Engineer — Eval & Automation

Rootstrap; Argentina; Remoto; ARS 106.019.000 - 166.601.000; Tempo pieno

Vantaggi offerti da questo lavoro

Certifications

Mentorship Program

Learning Bonus

Workation program

Gym benefits

Counseling services

Lunch reimbursements

Descrizione del lavoro

Rootstrap is seeking a Semi Senior QA Engineer to test AI-powered applications, including LLM-based features and RAG systems. You will design test plans, automate with Playwright, and build evaluation frameworks to measure quality and regressions.

The role emphasizes testing non-deterministic AI outputs, collaborating with engineers, and staying current with AI evaluation methods while enabling continuous QA improvements.

Competenze

• 3+ years of QA engineering experience.
• Hands-on mobile testing.
• Experience with Playwright automation.
• Experience testing AI-powered apps and LLM-based systems.
• Advanced English communication skills.

Mansioni

• Create detailed, comprehensive test plans and cases.
• Review specs with development teams to give feedback.
• Estimate, prioritize, plan, and coordinate testing activities.
• Identify, document, and track bugs.
• Design and maintain Playwright automation scripts.
• Perform regression testing when bugs are fixed.
• Track QA metrics and reports.
• Maintain eval suites to validate AI features and detect regressions.
• Test non-deterministic AI systems including LLMs and AI agents.
• Analyze AI outputs to identify failure patterns and drive improvements.
• Stay up-to-date with QA tools and AI evaluation techniques.
• Investigate defect causes with the dev team and contribute to preventive solutions.
• Collaborate on internal initiatives.

Conoscenze

QA testing

AI testing

Eval engineering

Automation testing

Automation scripting

Formazione

BSc in CS/SE

Strumenti

Playwright

LangSmith

OpenEvals

Arize Phoenix

CI/CD pipelines

Descrizione del lavoro

Rootstrap is seeking a Semi Senior QA Engineer to test AI-powered applications, including LLM-based features and RAG systems. You will design test plans, automate with Playwright, and build evaluation frameworks to measure quality and regressions.

The role emphasizes testing non-deterministic AI outputs, collaborating with engineers, and staying current with AI evaluation methods while enabling continuous QA improvements.