QA Engineer (AI & Feature Testing)

Solvd; United States; Remote; USD 110,000 - 160,000; Full time

Job summary

Solvd Inc. is seeking a Senior QA Engineer to own quality for AI-powered and traditional features across web, mobile, and API for a leading personal finance platform. You will focus on testing AI/LLM-integrated components and ensure robust validation in production environments.

The role requires deep experience in eval-based and deterministic testing, Playwright, and strong QA practices. You will work with a global team and align LATAM hours to collaborate effectively.

Qualifications

• 5–7 years of experience in software testing across web, mobile, and API.
• Hands-on experience with Playwright in a production testing environment.
• Hands-on AI feature testing experience validating AI/LLM-integrated products.
• Clear understanding of eval vs. deterministic testing and human-in-the-loop evaluations.
• Strong understanding of web and mobile app development lifecycles including iOS and Android.
• Proven experience in integration, regression, performance/load, and UAT.
• Hands-on knowledge of browser DevTools and log monitoring.
• Experience with test management systems (e.g., Jira, Confluence).
• Experience with HTML proxy tools like Charles Proxy or Fiddler.

Responsibilities

• Plan, design, and execute test cases for AI-powered and traditional features.
• Analyze AI model behavior specs, prompts, and edge-case scenarios.
• Convert requirements into test plans, cases, and scripts.
• Perform manual feature testing across platforms and verify fixes.
• Create clear defect reports and document AI outputs and regressions.
• Develop QA practices for emerging technologies and share best practices.
• Become a subject matter expert in core systems and reports.
• Collaborate with engineers, IT, and global teams with LATAM hours.

Skills

Playwright

AI testing

Eval vs deterministic testing

Web/mobile/API testing

Defect reporting

Browser DevTools

Test management

Education

Bachelor’s degree in Computer Science

Tools

Jira

Confluence

Charles Proxy

Fiddler

Xcode logs

ADB logs

Job description

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

We are looking for a Senior QA Engineer to join a team building and validating AI-powered and traditional features for a leading personal finance platform. You’ll own quality across web, mobile, and API — with a particular focus on testing AI/LLM-integrated components where standard deterministic methods don’t apply.

This role requires someone who understands the fundamental difference between testing software that has a right answer and evaluating AI outputs that have a probability distribution — and who has done both in production.

What you’ll do

• Plan, design, and execute test cases for AI-powered components and traditional features, applying the appropriate methodology — eval-based or deterministic — based on the target behavior.

• Analyze traditional functional requirements as well as AI model behavior specifications, prompt outputs, and edge-case scenarios.

• Turn business requirements, user flows, and business process procedures into comprehensive test plans, test cases, and test scripts.

• Perform manual feature testing across platforms and verify bug fixes.

• Create clear, detailed defect reports and document unexpected AI model responses, hallucinations, and performance regressions.

• Assist in developing QA practices for emerging technologies including AI frameworks, and disseminate best practices across the team.

• Become a subject matter expert in core systems, interfaces, business processes, and reporting tools.

• Partner closely with functional teams, engineers, and IT in a global environment with aligned LATAM working hours.

Required skills & experience

• 5–7 years of experience in software testing across web, mobile, and API.

• Hands‑on experience with Playwright in a production testing environment — this is a primary requirement for this role.

• Hands‑on AI feature testing experience — validating AI/LLM-integrated products for accuracy, relevance, robustness against adversarial prompts, edge cases, and safety/toxicity.

• Clear understanding of eval vs. deterministic testing — the difference between pass/fail based on exact input/output and evaluating probabilistic AI outputs using human-in-the-loop evaluations, score metrics, benchmark datasets, and semantic evaluation.

• Strong understanding of web and mobile application development lifecycles including iOS and Android specifics.

• Proven experience in integration, regression, performance/load, and UAT.

• Hands‑on knowledge of browser DevTools and log monitoring.

• Advanced proficiency in Jira and Confluence.

• Experience with test management systems.

Preferred qualifications

• Experience with HTML proxy tools such as Charles Proxy or Fiddler.

• Hands‑on knowledge of Xcode logs and ADB logs.

• Experience in the financial services or fintech domain.

• Bachelor’s degree in Computer Science or equivalent.

When you join Solvd, you'll…

• Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

• Be part of a global team with equal opportunities for collaboration across continents and cultures.

• Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Solvd is an equal opportunity employer.