Senior Product Software Engineer - AI Quality Engineering

C T Corporation System (DE)

Cape Girardeau (MO)

Presencial

USD 140.000 - 190.000

Jornada completa

14 días+

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

Wolters Kluwer Tax and Accounting seeks an experienced Senior AI Quality Engineer to own quality strategy and evaluation systems for AI-powered product experiences. You will architect, build, and maintain evaluation harnesses that measure retrieval accuracy, citation correctness, response quality, safety, performance, and alignment with customer intent.

You will lead automated testing, observability, and release readiness in CI/CD pipelines.

Formación

  • 5+ years of experience in quality engineering or related roles.
  • 2+ years testing AI/ML or non-deterministic systems in production.
  • Strong proficiency with Python test automation frameworks (e.g., Pytest, unittest).
  • Experience validating REST APIs, microservices, data pipelines, and AI workflows.

Responsabilidades

  • Own and implement quality engineering for enterprise-scale generative AI apps.
  • Design evaluation harnesses measuring accuracy, groundedness, safety, latency, and cost.
  • Develop automated regression suites tracking quality drift across models and configurations.
  • Lead quality gates for AI releases and advise on risk and readiness.
  • Mentor team members in AI evaluation methods and production-quality practices.

Conocimientos

Quality engineering
AI testing
Python
Pytest
unittest
REST APIs
CI/CD
Observability
Regression testing
Non-deterministic testing
SDET
Agile collaboration

Herramientas

Pytest
unittest

Descripción del empleo

We are seeking an experienced Senior AI Quality Engineer to own and implement quality strategy, evaluation systems, and reliability practices for AI-powered product experiences in Wolters Kluwer Tax and Accounting. In this role, you will architect, build, and maintain evaluation systems that measure retrieval accuracy, citation correctness, response quality, safety, performance, and overall alignment with customer intent. You will work hands-on across test automation, AI evaluation, APIs, data pipelines, observability, and release readiness while partnering with software engineers, product managers, UX, and stakeholders to ensure our generative AI systems are trustworthy, observable, secure, and ready for production use in professional tax and accounting workflows. This role goes beyond validating that features function as designed. You will own the practical quality engineering implementation for non-deterministic AI systems: defining how quality is measured, building automated evaluation and regression capabilities, implementing quality gates in delivery pipelines, driving release-readiness decisions, mentoring team members in AI quality practices, and establishing the standards customers can trust.

Key Responsibilities:
  • Own and implement the quality engineering approach for enterprise-scale generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines, and AI-driven user experiences.
  • Architect and implement evaluation harnesses that measure retrieval accuracy, citation correctness, answer relevance, groundedness, hallucination rate, safety, latency, cost, and end-to-end system behavior.
  • Design automated regression suites that detect quality drift across prompts, embeddings, chunking strategies, model versions, retrieval configuration, orchestration logic, and application workflows.
  • Partner with developers to validate AI system architecture, service integrations, API behavior, data ingestion, observability, resilience, security controls, and production readiness.
  • Lead root-cause analysis for AI quality failures using logs, traces, evaluation outputs, feedback signals, customer scenarios, and domain-specific acceptance criteria.
  • Define, implement, and maintain measurable quality gates for AI releases, and advise product and engineering leaders on risk, readiness, and tradeoffs across accuracy, reliability, latency, cost, and user trust.
  • Develop test automation in Python and related frameworks for APIs, services, chat workflows, data pipelines, and AI evaluation workflows within CI/CD pipelines.
  • Build and maintain observability practices and dashboards that track model behavior, retrieval performance, operational health, quality trends, and customer-impacting failure modes.
  • Contribute to responsible AI practices including bias and fairness checks, content safety validation, prompt injection testing, data privacy validation, and compliance-oriented evidence collection.
  • Collaborate with product managers, UX, subject matter experts, and stakeholders to translate customer workflows and business requirements into testable AI quality criteria.
  • Mentor engineers and quality team members in AI evaluation methods, non-deterministic testing strategies, automation design, and production-quality engineering practices.
  • Stay current with emerging AI quality, LLM evaluation, RAG assessment, and LLMOps practices through research, experimentation, and community engagement.
Key Requirements:
  • 5+ years of experience in quality engineering, SDET, software engineering, test automation, reliability engineering, or equivalent engineering roles.
  • 2+ years of hands-on experience testing, evaluating, or validating AI/ML, LLM, RAG, or other non-deterministic systems in production or production-like environments.
  • Strong proficiency with Python test automation frameworks such as Pytest, unittest, or equivalent, including experience building reusable test harnesses and regression suites.
  • Experience validating REST APIs, microservices, service integrations, data ingestion pipelines, and end-to-end web or chat-based workflows.
  • Practical understanding of RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality, and AI failure modes.
  • Experience defining quality metrics and success criteria where outputs are probabilistic rather than strictly deterministic.
  • Knowledge of CI/CD pipelines, Git-based workflows, automated regression testing, release gates, and modern engineering practices.
  • Experience with observability, logging, tracing, dashboards, and metrics used to monitor production behavior and diagnose system failures.
  • Ability to navigate ambiguity, decompose complex AI behaviors into testable risks, and communicate findings clearly to technical and non-technical stakeholders.
  • Strong collaboration skills in Agile environments, with demonstrated ability to influence engineering quality practices across product, design, development, and d
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Senior Product Software Engineer - AI Quality Engineering
Senior Product Software Engineer - AI Quality Engineering

Wolters Kluwer • Cape Girardeau (MO)

Presencial
USD 120.000 - 150.000
Senior AI Quality Engineer - GenAI Reliability & Trust
Senior AI Quality Engineer - GenAI Reliability & Trust

C T Corporation System (DE) • Cape Girardeau (MO)

Presencial
USD 140.000 - 190.000
V&V Engineer – AI-Driven Testing & Validation
V&V Engineer – AI-Driven Testing & Validation

Global Business Ser. 4u Inc. • Plano (TX)

Presencial
USD 120.000 - 180.000
Senior Staff SDET (Customer Engineering)
Senior Staff SDET (Customer Engineering)

Hims & Hers Health, Inc. • EE. UU.

A distancia
USD 180.000 - 240.000
Generous PTO
Full healthcare
401(k) with matching
+1
QA Automation Engineer - GenAI / LLM Testing (Healthcare)
QA Automation Engineer - GenAI / LLM Testing (Healthcare)

Luxoft • EE. UU.

A distancia
USD 110.000 - 150.000
AI Quality Architect
AI Quality Architect

EPAM Systems • EE. UU.

A distancia
USD 140.000 - 210.000
Lead QA Engineer
Lead QA Engineer

7Seventy • EE. UU.

Híbrido
USD 140.000 - 165.000
Equity
Lead Software Development Engineer in Test - AI
Lead Software Development Engineer in Test - AI

HighLevel • Texas

Híbrido
USD 140.000 - 200.000
Senior AI Systems Quality Engineer
Senior AI Systems Quality Engineer

Abacus Insights • EE. UU.

Presencial
USD 120.000 - 180.000
Senior AI Quality Engineer - GenAI & Observability
Senior AI Quality Engineer - GenAI & Observability

Wolters Kluwer • Cape Girardeau (MO)

Presencial
USD 120.000 - 150.000