Quality Assurance Engineer

Provenir

United States

Remote

USD 95,000 - 140,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Paid time off
Company holidays
Remote-friendly opportunities
Maternity/paternity leave

Job summary

Provenir is seeking a QA Engineer to ensure quality across the platform that handles decisioning at scale.

You will build end-to-end and API tests, automate in Python, and embed tests in CI/CD pipelines. The role touches graph data layers and AI/ML features, with opportunities to improve test architectures and reliability while working with cloud-based environments.

Qualifications

  • Bachelor’s degree in CS or related field.
  • Automated testing of web apps with Python (Playwright/Selenium).
  • Maintainable test architecture using Page Object Model.
  • Experience testing REST APIs with Postman/Swagger.
  • Knowledge of graph databases and queries.
  • Exposure to AI/ML-backed features and probabilistic systems.
  • Scripting beyond Python (Unix shell, Ruby).
  • CI/CD pipelines and automation experience.
  • SQL experience and process improvement mindset.

Responsibilities

  • Build end-to-end and component test suites with modern browser automation.
  • Shape coverage toward API and contract tests; UI for user journeys.
  • Embed quality gates in CI/CD with selective test runs.
  • Provision ephemeral containerized test environments via IaC.
  • Track flake rates and maintain test reliability with SLAs.
  • Use AI-assisted tooling for test authoring and coverage analysis.
  • Collaborate with engineers to shift-left testing and triage issues.
  • Validate graph data and analytics pipelines across Cypher/Gremlin/Gremlin-like queries.
  • Develop evaluation harnesses for ML model behavior and bias checks.

Skills

POM pattern
Python scripting
REST API testing
Graph databases
AI/ML feature testing
Unix shell scripting
CI/CD pipelines
SQL
Process improvement

Education

Bachelor's degree in Computer Science or related field

Tools

Postman
Swagger
Neo4j
Amazon Neptune
TigerGraph
JanusGraph

Job description

Who We Are Provenir is the unified Decision Intelligence Platform that gives enterprises full control over end- to-end customer decisioning — to manage risk, drive growth, and transform business outcomes. By consolidating data, AI models, intelligence, agents and governance into a single decisioning environment, Provenir empowers business teams to configure and evolve strategy directly, while maintaining enterprise-grade reliability and regulatory compliance. Trusted by 120+ institutions in 60+ countries, Provenir processes over 4 billion decisions annually — turning architectural coherence into sustained risk performance and measurable value. Ready to solve meaningful challenges, work with enterprise customers, and help shape the future of AI-powered decisioning? Join us.

The QA Engineer will be responsible for assuring the quality of our products, creating automated tests, and managing builds and continuous integration. You will automate tests for microservices and their related front ends, and manage infrastructure across development and production cloud environments. This role goes beyond conventional application testing. Our platform combines graph data stores, graph analytics, machine learning models, and agentic AI workflows — systems whose outputs are probabilistic, whose correctness is contextual, and whose decisions carry regulatory weight. You will help define what “passing” means for components that don’t return the same answer twice, and build the harnesses that hold them to it.

What You’ll Do
  • Core Test Automation - Build and maintain end-to-end and component test suites using modern browser automation with auto-waiting, trace-based debugging, and parallel execution as the default
  • Shape coverage around the test pyramid rather than the UI — API and contract tests as the primary safety net, with UI reserved for genuine user journeys
  • Embed quality gates directly in CI/CD pipelines — sharded parallel runs, risk-based test selection, and pass/fail criteria that block promotion without manual sign-off
  • Provision ephemeral, containerised test environments on demand through infrastructure‑as‑code, with synthetic and masked test data generated per run instead of maintained by hand
  • Treat test reliability as a product: track flake rates, quarantine and fix unstable tests, and hold suites to explicit stability and runtime SLAs
  • Use AI‑assisted tooling for test authoring, coverage gap analysis, and locator resilience — while owning the judgement call on what the generated tests are actually worth
  • Work shift‑left alongside engineers — contributing test coverage with the feature, not after it — and partner on triage to drive issues to root cause
  • Graph Data and Analytics - Design and automate validation for graph database layers — schema integrity, relationship correctness, and query behaviour across Cypher, Gremlin, or SPARQL
  • Build regression coverage for graph analytics outputs: pathfinding, centrality, community detection, and link analysis used in fraud‑ring identification and network risk scoring
  • Validate graph ingestion and transformation pipelines for data completeness, deduplication, and referential accuracy at scale
  • AI/ML and Agentic Systems - Develop evaluation harnesses for ML model behaviour — accuracy thresholds, drift detection, bias and fairness checks, and explainability outputs required for regulatory review
  • Build automated test suites for agentic workflows: tool‑calling correctness, multi‑step orchestration, context handling, failure recovery, and guardrail enforcement
  • Design assertion strategies for non‑deterministic outputs, including golden‑set comparison, semantic similarity scoring, and LLM‑as‑judge evaluation
  • Establish prompt and model regression testing so behavioural changes are caught before release
  • Validate feature pipelines and training/serving consistency in partnership with data science teams
  • Process and Platform - Incorporate best practices into dev/test/deploy processes and recommend improvements
  • Support application onboarding to the defined target operating model to achieve build and release automation
  • Document, track, and escalation issues as appropriate
What’s Required
Education

A Bachelor’s Degree in Computer Science or a related field.

Experience

Proven track record of extensive hands‑on experience in automated testing of web applications and services, including designing, developing, and executing automated tests in Python using a modern browser automation framework (Playwright, Selenium WebDriver, or equivalent).

Qualifications, Strengths and Skills
  • Demonstrated understanding of maintainable test architecture — Page Object Model or equivalent patterns — for end‑to‑end testing
  • Extensive hands‑on experience on Python scripting
  • Experience testing web services and REST APIs (Postman, Swagger)
  • Working knowledge of graph databases — Neo4j, Amazon Neptune, TigerGraph, JanusGraph or similar — including writing and validating graph queries
  • Exposure to testing AI/ML‑backed features, and a clear grasp of why probabilistic systems need different assertions than deterministic ones
  • Scripting experience beyond Python: Unix shell, Ruby, or similar
  • Continuous integration and pipeline‑based delivery (Jenkins, GitHub Actions, GitLab CI, or similar)
  • Experience with SQL
  • Experience recommending process improvements
Nice to Have
  • Hands‑on experience with agentic frameworks — LangChain, LangGraph, Model Context Protocol (MCP), or comparable orchestration tooling
  • Familiarity with LLM evaluation tooling and practices: golden datasets, rubric‑based scoring, hallucination and jailbreak testing, red‑teaming
  • Graph analytics libraries such as Neo4j GDS, Apache Spark GraphX, or NetworkX
  • Observability‑driven testing — synthetic monitoring, canary analysis, and validation through traces, logs and metrics
  • Docker, Kubernetes, Terraform, Ansible; microservices, container deployment and service orchestration
  • Cloud APIs (AWS, Azure, GCP)
  • JIRA for issue tracking and escalation

Our employees are our top priority; we offer comprehensive health and wellness plans.

  • paid time off
  • company holidays
  • flexible and remote‑friendly opportunities
  • maternity/paternity leave

At Provenir, we recognize that diversity and inclusion make our teams stronger. We are committed to equal employment opportunity and welcome everyone regardless of race, colour, ancestry, religion, national origin, age, sex, gender identity, sexual orientation, disability, marital status, domestic partner status, citizenship, or veteran status or medical condition. We encourage people from all backgrounds to apply.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Agentic AI Software Engineer
Agentic AI Software Engineer

Provenir • New York (NY)

Remote
USD 120,000 - 170,000
Health benefits
PTO & holidays
Remote-friendly
+1
QA Engineer
QA Engineer

Prudentia Sciences • Boston (MA)

Hybrid
PHP 6,897,000 - 8,777,000
Competitive salary
Equity option
Hybrid work
QA Engineer – Hybrid AI Biotech Platform
QA Engineer – Hybrid AI Biotech Platform

Prudentia Sciences • Boston (MA)

Hybrid
PHP 6,897,000 - 8,777,000
Competitive salary
Equity option
Hybrid work
Director of Quality Assurance
Director of Quality Assurance

Profisee • Alpharetta (GA)

On-site
USD 180,000 - 240,000
Senior Software QA Engineer
Senior Software QA Engineer

Qualifyze GmbH • United States

Hybrid
USD 56,000 - 67,000
Hybrid work mode
Annual development budget
Private health care
Principal Quality Engineer (AI-Assisted & Agentic Testing)
Principal Quality Engineer (AI-Assisted & Agentic Testing)

Modus Create • United States

Remote
USD 150,000 - 210,000
QA Tester
QA Tester

Peregrine Advisors LLC • United States

Remote
USD 100,000 - 140,000
Health insurance
401(k) with employer match
Unlimited PTO
+1
Senior QA Automation Engineer
Senior QA Automation Engineer

Proxify AB • United States

Remote
USD 120,000 - 170,000
Alliances Manager (AWS, Anthropic, GCP)
Alliances Manager (AWS, Anthropic, GCP)

Provectus • Rhode Island

On-site
USD 250,000 - 300,000
Unlimited vacation
Health, vision, and dental insurance
401(K) matching
+1
Alliances Manager (AWS, Anthropic, GCP)
Alliances Manager (AWS, Anthropic, GCP)

Provectus • New York (NY)

On-site
USD 180,000 - 260,000
Unlimited vacation
401(k) matching plan
Health, vision, and dental insurance