QA & AI Evaluation Engineer

FitNext Consulting; São Paulo; Hybrid; BRL 120,000 - 180,000; Full time

Benefits offered by this job

Flexible work hours

Job summary

FitNext Consulting is seeking a QA Engineer to build and own the quality function for a fast-growing AI-powered platform. You will create automated test coverage and develop ways to evaluate AI-generated outputs, while working with a small, highly technical team across Engineering and Product.

You will lead QA strategy, define release quality gates, and own testing for non-deterministic outputs in a fast-paced, early-stage environment.

Qualifications

• Minimum 5+ years in QA, SDET or Test Engineering.
• Experience owning automated test frameworks for production systems.
• Strong Python and/or TypeScript skills.
• Experience with modern E2E testing tools (Playwright, Cypress).
• Proficient API testing and CI/CD pipeline work.
• Experience with regression, integration, and end-to-end testing.
• Ability to manage flaky tests and false positives.
• Understanding release quality gates and test prioritization.
• Experience testing non-deterministic outputs.
• Professional to fluent English.

Responsibilities

• Build and own the company's QA function and test strategy.
• Develop automated test coverage for production products.
• Design evaluation methods for AI-generated outputs.
• Collaborate with Engineering, Product, and founding team on quality.

Skills

QA ownership

Automation frameworks

Python

TypeScript

Playwright

Cypress

API testing

CI/CD

Non-deterministic outputs

English fluency

Tools

Playwright

Cypress

Job description

We're working with a fast-growing Series A startup building an AI-powered platform for enterprise software delivery. The company is backed by leading investors and works with large global organizations on complex, high-impact projects.

You'll join a small, highly technical team and work closely with Engineering, Product, and the founding team.

The Challenge

We're looking for a QA Engineer to build and own the company's quality function.

This role goes beyond traditional QA. You'll be responsible for building automated test coverage while also developing ways to evaluate AI-generated and non-deterministic outputs.

You'll have significant ownership over how quality is defined, tested, and maintained across the product.

Must Have

• 5+ years of experience in QA, SDET, or Test Engineering
• Strong experience building and owning automated test frameworks for production products
• Strong experience with Python and/or TypeScript
• Experience with modern E2E testing frameworks such as Playwright, Cypress, or equivalent
• Strong API testing experience
• Hands-on CI/CD experience, including building and maintaining test pipelines
• Experience with regression, integration, and end-to-end testing
• Experience identifying and managing flaky tests and false positives
• Strong understanding of test prioritization, severity, and release quality gates
• Experience testing systems with non-deterministic outputs
• Professional-to-fluent spoken English

Nice to Have

• Experience testing LLM-powered or agentic features
• Experience designing evaluation frameworks, golden datasets, or scoring criteria
• Experience with hallucination detection or human-in-the-loop workflows
• Experience with SAP, ERP, CRM, or other enterprise software
• Experience with SAP testing tools such as Tricentis or Worksoft
• Experience with UAT and enterprise software testing cycles
• Performance, load, or security testing experience
• Ability to contribute code and fix bugs, not only identify them

Soft Skills

• High ownership and autonomy
• Strong judgment and prioritization
• Comfortable working without an established playbook
• Strong attention to quality and detail
• Comfortable challenging decisions when quality is at risk
• Strong communication and collaboration skills
• Comfortable working in a fast-paced, early-stage environment
• Direct access to Engineering, Product, and the founding team
• High ownership over the QA function and quality standards
• Opportunity to work on challenging AI and enterprise software testing problems
• Opportunity to build QA processes and infrastructure from the ground up