QA Engineer (AI & Feature Testing)

Jobgether; United States; Remote; USD 90,000 - 140,000; Full time

Benefits offered by this job

100% remote work

International collaboration

Meaningful AI projects

Professional development

Continuous learning

Inclusive workplace

Responsible AI focus

Career growth opportunities

Job summary

Jobgether is seeking a Senior QA Engineer to lead testing across web, mobile, and API platforms, with a strong focus on AI-powered features. You will design test strategies, validate AI outputs, and drive quality across continuous release cycles.

The ideal candidate has 5–7 years of hands-on testing experience, proven Playwright expertise, and familiarity with AI evaluation methods. Remote-friendly role with collaboration across international teams.

Qualifications

• 5–7 years of hands-on software testing across web, mobile, and APIs.
• Proven Playwright production experience for software testing.
• Experience testing AI-powered products and LLM-integrated features.

Responsibilities

• Plan, design, and execute test plans, scripts, and cases across web, mobile, and API environments.
• Validate AI-driven functionality for accuracy, relevance, and safety.
• Conduct regression, integration, performance/load, and UAT testing.
• Identify, document, and track defects with clear repro steps and evidence.
• Collaborate with engineers and product stakeholders to ensure quality across releases.

Skills

Playwright

AI & LLM testing

Testing strategies

Web & mobile testing

Debugging & diagnostics

Education

Bachelor's degree in Computer Science or equivalent

Tools

Jira

Confluence

HTTP debugging tools (Charles Proxy / Fiddler)

Xcode logs / ADB logs

Job description

Accountabilities

• Plan and execute comprehensive testing: Design, develop, and execute test plans, test cases, and test scripts for traditional software features and AI-powered components across web, mobile, and API environments.
• Validate AI-driven functionality: Test AI and LLM-integrated features for accuracy, relevance, consistency, robustness, and safety, identifying hallucinations, unexpected responses, and failures in edge cases.
• Apply appropriate testing methodologies: Determine when to use deterministic pass/fail testing and when to apply evaluation-based approaches for probabilistic AI outputs, ensuring test strategies reflect the expected behavior of each feature.
• Evaluate AI model performance: Assess AI-generated outputs using appropriate evaluation metrics, benchmark datasets, semantic assessments, and human-in-the-loop review processes.
• Translate requirements into test scenarios: Analyze business requirements, user journeys, process documentation, and AI behavior specifications to develop comprehensive and effective testing coverage.
• Conduct manual feature testing: Execute functional tests across web and mobile applications, verify expected behavior, validate fixes, and ensure new releases meet quality expectations.
• Perform regression and integration testing: Validate interactions between application components and APIs while conducting regression, performance/load, and user acceptance testing (UAT).
• Identify and document defects: Create clear, detailed bug reports that include reproducible steps, expected and actual results, supporting evidence, and relevant technical information.
• Investigate technical issues: Use browser developer tools, application logs, and debugging utilities to investigate failures, trace unexpected behavior, and support efficient root-cause analysis.
• Support mobile quality assurance: Validate mobile application behavior across relevant iOS and Android environments, considering platform-specific functionality and potential compatibility issues.
• Improve QA practices: Contribute to testing standards, methodologies, and best practices for emerging technologies, including AI frameworks and LLM-based applications.
• Maintain testing documentation: Use Jira, Confluence, and test management systems to track defects, organize test cases, document results, and maintain clear project records.
• Collaborate across teams: Partner with engineers, functional specialists, product stakeholders, and IT teams to resolve issues, align testing priorities, and ensure quality throughout the development lifecycle.
• Develop subject-matter expertise: Build a strong understanding of core systems, application interfaces, business processes, and reporting tools to strengthen test coverage and improve the reliability of delivered solutions.
• Promote responsible AI quality: Help ensure AI-powered features meet appropriate standards for reliability, robustness, safety, and user trust.

Requirements

• Professional experience: 5–7 years of hands-on software testing experience across web applications, mobile platforms, and APIs.
• Playwright expertise: Proven production experience using Playwright for software testing. This is a primary requirement for the position.
• AI and LLM testing: Practical experience testing AI-powered products or LLM-integrated features, including evaluating accuracy, relevance, robustness, adversarial prompt handling, edge cases, and safety or toxicity risks.
• Testing methodology: Clear understanding of the differences between deterministic testing, which relies on defined inputs and expected outputs, and evaluation-based testing for probabilistic AI systems.
• AI evaluation techniques: Familiarity with human-in-the-loop evaluations, scoring metrics, benchmark datasets, semantic evaluation, and other approaches for assessing AI-generated responses.
• Software testing fundamentals: Demonstrated experience with functional, integration, regression, performance/load, and user acceptance testing.
• Web and mobile development knowledge: Strong understanding of application development lifecycles, including web technologies and platform-specific considerations for iOS and Android.
• Debugging and diagnostics: Hands-on experience with browser developer tools, application logs, and technical investigation techniques for identifying and resolving defects.
• Test management and documentation: Advanced proficiency in Jira and Confluence, along with experience using test management systems to organize test cases, execution results, and defect tracking.
• Analytical thinking: Strong attention to detail and the ability to investigate complex issues, recognize unexpected patterns, and distinguish between software defects and acceptable variations in AI-generated outputs.
• Communication skills: Ability to document technical findings clearly, communicate quality risks effectively, and collaborate with cross-functional stakeholders.
• Teamwork and adaptability: Comfortable working in a distributed, international environment, coordinating with engineering and functional teams across different locations.
• Preferred technical experience: Familiarity with HTTP proxy and debugging tools such as Charles Proxy or Fiddler, as well as Xcode logs and Android Debug Bridge (ADB) logs.
• Preferred industry experience: Previous experience in financial services, fintech, or personal finance technology is an advantage.
• Preferred education: A bachelor's degree in Computer Science or an equivalent technical qualification.
• Quality and ethics mindset: Commitment to continuous improvement, responsible AI practices, and delivering reliable, secure, and user-focused software.

Benefits

• 100% remote work: Work remotely from Brazil while collaborating with an international technology team.
• International collaboration: Work with colleagues across countries and cultures, contributing to projects with global reach.
• Meaningful AI projects: Help build and validate real-world AI-powered solutions and contribute to the evolution of AI-enabled products.
• Professional development: Strengthen your expertise in software quality assurance, advanced testing methodologies, and AI/LLM evaluation.
• Continuous learning: Join an environment that encourages technical development, knowledge sharing, and experimentation with emerging technologies.
• Inclusive workplace: Contribute to a collaborative culture that values diversity, equal opportunity, and different perspectives.
• Responsible AI focus: Participate in work that emphasizes ethical AI standards, product reliability, and user trust.
• Career growth opportunities: Expand your technical capabilities by working on challenging projects spanning software engineering, AI, and enterprise technology.