QA Engineer (AI + Python)

Apreel

United States

On-site

USD 90,000 - 130,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Apreel is seeking an AI & Backend QA Engineer to own the quality of model outputs and backend APIs.

You will build evaluation practices, develop regression detection, and implement contract testing with Pact, in collaboration with the US engineering team. The role emphasizes localization of errors across model, service, and input data.

Qualifications

  • 3+ years of technical QA/test automation including API testing on production.
  • Experience with Python and pytest.
  • Strong HTTP/REST, JSON and auth token handling.
  • Contract testing with Pact is a hard requirement.
  • Security-aware API testing including boundary and malformed input testing.
  • Comfort with non-determinism: validate properties/contracts rather than exact outputs.
  • LLM evaluation including datasets, golden sets, and regression detection.
  • Prompt engineering: modify prompts and measure effects; watch for hallucinations.
  • Backend/API testing across services; isolate model vs. data vs. infrastructure.
  • Proficiency with Postman and Swagger/OpenAPI in daily work.
  • Git in team workflow with PRs and reviews.
  • TestRail for test-case management and runs.
  • Mobile testing for Android/iOS contexts and store builds.

Responsibilities

  • Define and own QA strategy for AI core: golden sets, metrics, and regression detection.
  • Develop and maintain API test suites, including Pact contract tests.
  • Test non-deterministic AI outputs and backend APIs with strong evidence-based checks.
  • Collaborate with frontend/mobile teams to ensure end-to-end quality across surfaces.

Skills

3+ years in technical QA/test autom
Python testing (pytest)
HTTP/REST APIs
Contract testing with Pact
Security-aware API testing
Non-deterministic test strategies
LLM evaluation datasets & metrics
Prompt engineering
Backend/API testing
Postman / Swagger / OpenAPI
Git workflows
TestRail for test management
Mobile app testing (Android/iOS)

Tools

Postman
Swagger/OpenAPI
Pact
Appium
LangGraph/LangSmith

Job description

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:
  • AI core — a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this
  • role's primary surface
  • Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it
  • Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that
  • members use
Kogo poszukujemy?
Must-have skills:
  • 3+ years in technical QA/test automation, incl. API testing on production
  • Python — writing/maintaining tests, pytest, fixtures
  • HTTP/API fundamentals — REST, JSON, status codes, headers, auth tokens
  • Contract testing with Pact — consumer-driven contracts (hard requirement)
  • Security-aware API testing — auth boundaries, token handling, malformed/hostile input
  • Comfort with non-determinism — assert on properties/contracts, not exact output
  • LLM evaluation — datasets/golden sets, response metrics, detect regressions across prompts/models
  • Prompt engineering — modify prompts and measure effects; aware of hallucinations and prompt injection
  • Backend savvy — HTTP/REST, JSON, tokens, logs, DB queries; isolate model vs service vs input issues
  • Postman and Swagger/OpenAPI daily
  • Git in team workflow — branches, PRs, reviews
  • TestRail for test-case management and runs
  • Mobile product context (Android/iOS) — testing app features, release timing, store builds
Nice-to-haves:
  • Langfuse or equivalent (LangSmith, Phoenix) — traces and attached evaluations
  • LangGraph — test agents as state graphs (single node and full flow)
  • CI/CD — run evaluations in pipelines and quality gates on PRs
  • Python with Appium — mobile automation exposure alongside API work
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

QA Engineer — Web & Mobile
QA Engineer — Web & Mobile

Physitrack • United States

Remote
USD 57,000 - 68,000
QA Automation Engineer
QA Automation Engineer

GCS Recruitment • United States

Remote
USD 120,000 - 170,000
Quality Assurance Engineer
Quality Assurance Engineer

Napster • United States

Hybrid
USD 120,000 - 170,000
QA Engineer
QA Engineer

Npv • Paris (TX)

Hybrid
USD 59,000 - 89,000
Hybrid work Paris/London
Relocation package to Paris
Medical insurance in France
+3
QA Engineer
QA Engineer

WebSenor Ltd • United States

Remote
USD 70,000 - 110,000
Lead Python QA Automation Engineer
Lead Python QA Automation Engineer

EPAM Systems • United States

Remote
USD 120,000 - 150,000
QA Lead
QA Lead

intent • United States

Remote
USD 58,000 - 80,000
Remote work
Paid leave 26 days
Paid sick leave 10 days
+6
QA Engineer
QA Engineer

5Blue Software • United States

Remote
USD 90,000 - 130,000
Remote work
English classes
Health insurance
+5
Quality Assurance Engineer
Quality Assurance Engineer

Napster Corp. • United States

Remote
USD 90,000 - 120,000
AI-Augmented QA Engineer
AI-Augmented QA Engineer

Pavago • United States

Remote
USD 90,000 - 140,000